Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Jeremy Stanley]

On 2017-05-18 18:04:35 -0400 (-0400), Paul Belanger wrote:
[...]
> if we decide to publish to docker, I don't think we'd push
> directly. Maybe push to our docker registry then mirror to docker
> hub. That is something we can figure out a little later.
[...]

Ideally by iterating on https://review.openstack.org/447524 where
the details for what that would look like are being hashed out.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Paul Belanger]

On Thu, May 18, 2017 at 09:34:44AM -0700, Michał Jastrzębski wrote:
> >> Issue with that is
> >>
> >> 1. Apache served is harder to use because we want to follow docker API
> >> and we'd have to reimplement it
> >
> > No, the idea is apache is transparent, for now we have been using proxypass
> > module in apache.  I think what Doug was mentioning was have a primary 
> > docker
> > registery, with is RW for a publisher, then proxy it to regional mirrors as 
> > RO.
> 
> That would also work, yes
> 
> >> 2. Running registry is single command
> >>
> > I've seen this mentioned a few times before, just because it is one command 
> > or
> > 'simple' to do, doesn't mean we want to or can.  Currently our 
> > infrastructure is
> > complicated, for various reasons.  I am sure we'll get to the right 
> > technical
> > solution for making jobs happy. Remember our infrastructure spans 6 clouds 
> > and 15
> > regions and want to make sure it is done correctly.
> 
> And that's why we discussed dockerhub. Remember that I was willing to
> implement proper registry, but we decided to go with dockerhub simply
> because it puts less stress on both infra and infra team. And I
> totally agree with that statement. Dockerhub publisher + apache
> caching was our working idea.
> 
yes, we still want to implement a docker registry for openstack, maybe for
testing, maybe for production. From the technical side, we have a good handle
now how that would look.  However, even if we decide to publish to docker, I
don't think we'd push directly. Maybe push to our docker registry then mirror to
docker hub. That is something we can figure out a little later.

> >> 3. If we host in in infra, in case someone actually uses it (there
> >> will be people like that), that will eat up lot of network traffic
> >> potentially
> >
> > We can monitor this and adjust as needed.
> >
> >> 4. With local caching of images (working already) in nodepools we
> >> loose complexity of mirroring registries across nodepools
> >>
> >> So bottom line, having dockerhub/quay.io is simply easier.
> >>
> > See comment above.
> >
> >> > Doug
> >> >
> >> > __
> >> > OpenStack Development Mailing List (not for usage questions)
> >> > Unsubscribe: 
> >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 18 May 2017 at 08:03, Paul Belanger  wrote:
> On Tue, May 16, 2017 at 02:11:18PM +, Sam Yaple wrote:
>> I would like to bring up a subject that hasn't really been discussed in
>> this thread yet, forgive me if I missed an email mentioning this.
>>
>> What I personally would like to see is a publishing infrastructure to allow
>> pushing built images to an internal infra mirror/repo/registry for
>> consumption of internal infra jobs (deployment tools like kolla-ansible and
>> openstack-ansible). The images built from infra mirrors with security
>> turned off are perfect for testing internally to infra.
>>
> Zuulv3 should have a little with this, it will allow for DAG graph for jobs,
> which means the top level job could be an image build then all jobs below can
> now consume said image.  The steps we are still working on is artifact 
> handling
> but long term, it should be possible for the testing jobs to setup the dynamic
> infrastructure needed themselves.
>
>> If you build images properly in infra, then you will have an image that is
>> not security checked (no gpg verification of packages) and completely
>> unverifiable. These are absolutely not images we want to push to
>> DockerHub/quay for obvious reasons. Security and verification being chief
>> among them. They are absolutely not images that should ever be run in
>> production and are only suited for testing. These are the only types of
>> images that can come out of infra.
>>
> We disable gpg for Ubuntu packaging for a specific reason, most this is 
> because
> our APT repos are not official mirrors of upstream. We regenerate indexes 
> every
> 2 hours as not to break long running jobs.  We have talked in the past of 
> fixing
> this, but it requires openstack-infra to move to a new mirroring tool for APT.

So idea to solve this particular problem goes like this:

Publish job is not a change-driven, it'll be periodical (24h?) during
low time. Then in this job we can turn off using infra mirrors and
just use upstream signed.

That being said, all the technical issues we saw so far (unless I'm
missing something) are solvable and we (kolla community) would love to
do all the heavy lifting to solve it. We need to wait for TC to
resolve non-technical issues before we can proceed tho.

>> Thanks,
>> SamYaple
>>
>> On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski 
>> wrote:
>>
>> > On 16 May 2017 at 06:22, Doug Hellmann  wrote:
>> > > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
>> > >> Flavio Percoco wrote:
>> > >> > From a release perspective, as Doug mentioned, we've avoided
>> > releasing projects
>> > >> > in any kind of built form. This was also one of the concerns I raised
>> > when
>> > >> > working on the proposal to support other programming languages. The
>> > problem of
>> > >> > releasing built images goes beyond the infrastructure requirements.
>> > It's the
>> > >> > message and the guarantees implied with the built product itself that
>> > are the
>> > >> > concern here. And I tend to agree with Doug that this might be a
>> > problem for us
>> > >> > as a community. Unfortunately, putting your name, Michal, as contact
>> > point is
>> > >> > not enough. Kolla is not the only project producing container images
>> > and we need
>> > >> > to be consistent in the way we release these images.
>> > >> >
>> > >> > Nothing prevents people for building their own images and uploading
>> > them to
>> > >> > dockerhub. Having this as part of the OpenStack's pipeline is a
>> > problem.
>> > >>
>> > >> I totally subscribe to the concerns around publishing binaries (under
>> > >> any form), and the expectations in terms of security maintenance that it
>> > >> would set on the publisher. At the same time, we need to have images
>> > >> available, for convenience and testing. So what is the best way to
>> > >> achieve that without setting strong security maintenance expectations
>> > >> for the OpenStack community ? We have several options:
>> > >>
>> > >> 1/ Have third-parties publish images
>> > >> It is the current situation. The issue is that the Kolla team (and
>> > >> likely others) would rather automate the process and use OpenStack
>> > >> infrastructure for it.
>> > >>
>> > >> 2/ Have third-parties publish images, but through OpenStack infra
>> > >> This would allow to automate the process, but it would be a bit weird to
>> > >> use common infra resources to publish in a private repo.
>> > >>
>> > >> 3/ Publish transient (per-commit or daily) images
>> > >> A "daily build" (especially if you replace it every day) would set
>> > >> relatively-limited expectations in terms of maintenance. It would end up
>> > >> picking up security updates in upstream layers, even if not immediately.
>> > >>
>> > >> 4/ Publish images and own them
>> > >> Staff release / VMT / stable team in a way that lets us properly own
>> > >> those images and publish them 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

>> Issue with that is
>>
>> 1. Apache served is harder to use because we want to follow docker API
>> and we'd have to reimplement it
>
> No, the idea is apache is transparent, for now we have been using proxypass
> module in apache.  I think what Doug was mentioning was have a primary docker
> registery, with is RW for a publisher, then proxy it to regional mirrors as 
> RO.

That would also work, yes

>> 2. Running registry is single command
>>
> I've seen this mentioned a few times before, just because it is one command or
> 'simple' to do, doesn't mean we want to or can.  Currently our infrastructure 
> is
> complicated, for various reasons.  I am sure we'll get to the right technical
> solution for making jobs happy. Remember our infrastructure spans 6 clouds 
> and 15
> regions and want to make sure it is done correctly.

And that's why we discussed dockerhub. Remember that I was willing to
implement proper registry, but we decided to go with dockerhub simply
because it puts less stress on both infra and infra team. And I
totally agree with that statement. Dockerhub publisher + apache
caching was our working idea.

>> 3. If we host in in infra, in case someone actually uses it (there
>> will be people like that), that will eat up lot of network traffic
>> potentially
>
> We can monitor this and adjust as needed.
>
>> 4. With local caching of images (working already) in nodepools we
>> loose complexity of mirroring registries across nodepools
>>
>> So bottom line, having dockerhub/quay.io is simply easier.
>>
> See comment above.
>
>> > Doug
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Paul Belanger]

On Tue, May 16, 2017 at 02:11:18PM +, Sam Yaple wrote:
> I would like to bring up a subject that hasn't really been discussed in
> this thread yet, forgive me if I missed an email mentioning this.
> 
> What I personally would like to see is a publishing infrastructure to allow
> pushing built images to an internal infra mirror/repo/registry for
> consumption of internal infra jobs (deployment tools like kolla-ansible and
> openstack-ansible). The images built from infra mirrors with security
> turned off are perfect for testing internally to infra.
> 
Zuulv3 should have a little with this, it will allow for DAG graph for jobs,
which means the top level job could be an image build then all jobs below can
now consume said image.  The steps we are still working on is artifact handling
but long term, it should be possible for the testing jobs to setup the dynamic
infrastructure needed themselves.

> If you build images properly in infra, then you will have an image that is
> not security checked (no gpg verification of packages) and completely
> unverifiable. These are absolutely not images we want to push to
> DockerHub/quay for obvious reasons. Security and verification being chief
> among them. They are absolutely not images that should ever be run in
> production and are only suited for testing. These are the only types of
> images that can come out of infra.
> 
We disable gpg for Ubuntu packaging for a specific reason, most this is because
our APT repos are not official mirrors of upstream. We regenerate indexes every
2 hours as not to break long running jobs.  We have talked in the past of fixing
this, but it requires openstack-infra to move to a new mirroring tool for APT.

> Thanks,
> SamYaple
> 
> On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski 
> wrote:
> 
> > On 16 May 2017 at 06:22, Doug Hellmann  wrote:
> > > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
> > >> Flavio Percoco wrote:
> > >> > From a release perspective, as Doug mentioned, we've avoided
> > releasing projects
> > >> > in any kind of built form. This was also one of the concerns I raised
> > when
> > >> > working on the proposal to support other programming languages. The
> > problem of
> > >> > releasing built images goes beyond the infrastructure requirements.
> > It's the
> > >> > message and the guarantees implied with the built product itself that
> > are the
> > >> > concern here. And I tend to agree with Doug that this might be a
> > problem for us
> > >> > as a community. Unfortunately, putting your name, Michal, as contact
> > point is
> > >> > not enough. Kolla is not the only project producing container images
> > and we need
> > >> > to be consistent in the way we release these images.
> > >> >
> > >> > Nothing prevents people for building their own images and uploading
> > them to
> > >> > dockerhub. Having this as part of the OpenStack's pipeline is a
> > problem.
> > >>
> > >> I totally subscribe to the concerns around publishing binaries (under
> > >> any form), and the expectations in terms of security maintenance that it
> > >> would set on the publisher. At the same time, we need to have images
> > >> available, for convenience and testing. So what is the best way to
> > >> achieve that without setting strong security maintenance expectations
> > >> for the OpenStack community ? We have several options:
> > >>
> > >> 1/ Have third-parties publish images
> > >> It is the current situation. The issue is that the Kolla team (and
> > >> likely others) would rather automate the process and use OpenStack
> > >> infrastructure for it.
> > >>
> > >> 2/ Have third-parties publish images, but through OpenStack infra
> > >> This would allow to automate the process, but it would be a bit weird to
> > >> use common infra resources to publish in a private repo.
> > >>
> > >> 3/ Publish transient (per-commit or daily) images
> > >> A "daily build" (especially if you replace it every day) would set
> > >> relatively-limited expectations in terms of maintenance. It would end up
> > >> picking up security updates in upstream layers, even if not immediately.
> > >>
> > >> 4/ Publish images and own them
> > >> Staff release / VMT / stable team in a way that lets us properly own
> > >> those images and publish them officially.
> > >>
> > >> Personally I think (4) is not realistic. I think we could make (3) work,
> > >> and I prefer it to (2). If all else fails, we should keep (1).
> > >>
> > >
> > > At the forum we talked about putting test images on a "private"
> > > repository hosted on openstack.org somewhere. I think that's option
> > > 3 from your list?
> > >
> > > Paul may be able to shed more light on the details of the technology
> > > (maybe it's just an Apache-served repo, rather than a full blown
> > > instance of Docker's service, for example).
> >
> > Issue with that is
> >
> > 1. Apache served is harder to use because we want to follow docker API
> > 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Paul Belanger]

On Tue, May 16, 2017 at 06:57:04AM -0700, Michał Jastrzębski wrote:
> On 16 May 2017 at 06:22, Doug Hellmann  wrote:
> > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
> >> Flavio Percoco wrote:
> >> > From a release perspective, as Doug mentioned, we've avoided releasing 
> >> > projects
> >> > in any kind of built form. This was also one of the concerns I raised 
> >> > when
> >> > working on the proposal to support other programming languages. The 
> >> > problem of
> >> > releasing built images goes beyond the infrastructure requirements. It's 
> >> > the
> >> > message and the guarantees implied with the built product itself that 
> >> > are the
> >> > concern here. And I tend to agree with Doug that this might be a problem 
> >> > for us
> >> > as a community. Unfortunately, putting your name, Michal, as contact 
> >> > point is
> >> > not enough. Kolla is not the only project producing container images and 
> >> > we need
> >> > to be consistent in the way we release these images.
> >> >
> >> > Nothing prevents people for building their own images and uploading them 
> >> > to
> >> > dockerhub. Having this as part of the OpenStack's pipeline is a problem.
> >>
> >> I totally subscribe to the concerns around publishing binaries (under
> >> any form), and the expectations in terms of security maintenance that it
> >> would set on the publisher. At the same time, we need to have images
> >> available, for convenience and testing. So what is the best way to
> >> achieve that without setting strong security maintenance expectations
> >> for the OpenStack community ? We have several options:
> >>
> >> 1/ Have third-parties publish images
> >> It is the current situation. The issue is that the Kolla team (and
> >> likely others) would rather automate the process and use OpenStack
> >> infrastructure for it.
> >>
> >> 2/ Have third-parties publish images, but through OpenStack infra
> >> This would allow to automate the process, but it would be a bit weird to
> >> use common infra resources to publish in a private repo.
> >>
> >> 3/ Publish transient (per-commit or daily) images
> >> A "daily build" (especially if you replace it every day) would set
> >> relatively-limited expectations in terms of maintenance. It would end up
> >> picking up security updates in upstream layers, even if not immediately.
> >>
> >> 4/ Publish images and own them
> >> Staff release / VMT / stable team in a way that lets us properly own
> >> those images and publish them officially.
> >>
> >> Personally I think (4) is not realistic. I think we could make (3) work,
> >> and I prefer it to (2). If all else fails, we should keep (1).
> >>
> >
> > At the forum we talked about putting test images on a "private"
> > repository hosted on openstack.org somewhere. I think that's option
> > 3 from your list?
> >
> > Paul may be able to shed more light on the details of the technology
> > (maybe it's just an Apache-served repo, rather than a full blown
> > instance of Docker's service, for example).
> 
> Issue with that is
> 
> 1. Apache served is harder to use because we want to follow docker API
> and we'd have to reimplement it

No, the idea is apache is transparent, for now we have been using proxypass
module in apache.  I think what Doug was mentioning was have a primary docker
registery, with is RW for a publisher, then proxy it to regional mirrors as RO.

> 2. Running registry is single command
>
I've seen this mentioned a few times before, just because it is one command or
'simple' to do, doesn't mean we want to or can.  Currently our infrastructure is
complicated, for various reasons.  I am sure we'll get to the right technical
solution for making jobs happy. Remember our infrastructure spans 6 clouds and 
15
regions and want to make sure it is done correctly.

> 3. If we host in in infra, in case someone actually uses it (there
> will be people like that), that will eat up lot of network traffic
> potentially

We can monitor this and adjust as needed.

> 4. With local caching of images (working already) in nodepools we
> loose complexity of mirroring registries across nodepools
> 
> So bottom line, having dockerhub/quay.io is simply easier.
> 
See comment above.

> > Doug
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Bogdan Dobrelya]

On 16.05.2017 20:57, Michał Jastrzębski wrote:
> On 16 May 2017 at 11:49, Doug Hellmann  wrote:
>> Excerpts from Michał Jastrzębski's message of 2017-05-16 11:38:19 -0700:
>>> On 16 May 2017 at 11:27, Doug Hellmann  wrote:
 Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700:
> So another consideration. Do you think whole rule of "not building
> binares" should be reconsidered? We are kind of new use case here. We
> aren't distro but we are packagers (kind of). I don't think putting us
> on equal footing as Red Hat, Canonical or other companies is correct
> here.
>
> K8s is something we want to work with, and what we are discussing is
> central to how k8s is used. K8s community creates this culture of
> "organic packages" built by anyone, most of companies/projects already
> have semi-official container images and I think expectations on
> quality of these are well...none? You get what you're given and if you
> don't agree, there is always way to reproduce this yourself.
>
> [Another huge snip]
>

 I wanted to have the discussion, but my position for now is that
 we should continue as we have been and not change the policy.

 I don't have a problem with any individual or group of individuals
 publishing their own organic packages. The issue I have is with
 making sure it is clear those *are* "organic" and not officially
 supported by the broader community. One way to do that is to say
 they need to be built somewhere other than on our shared infrastructure.
 There may be other ways, though, so I'm looking for input on that.
>>>
>>> What I was trying to say here is, current discussion aside, maybe we
>>> should revise this "not supported by broader community" rule. They may
>>> very well be supported to a certain point. Support is not just yes or
>>> no, it's all the levels in between. I think we can afford *some* level
>>> of official support, even if that some level means best effort made by
>>> community. If Kolla community, not an individual like myself, would
>>> like to support these images best to our ability, why aren't we
>>> allowed? As long as we are crystal clear what is scope of our support,
>>> why can't we do it? I think we've already proven that it's going to be
>>> tremendously useful for a lot of people, even in a shape we discuss
>>> today, that is "best effort, you still need to validate it for
>>> yourself"...
>>
>> Right, I understood that. So far I haven't heard anything to change
>> my mind, though.
>>
>> I think you're underestimating the amount of risk you're taking on
>> for yourselves and by extension the rest of the community, and
>> introducing to potential consumers of the images, by promising to
>> support production deployments with a small team of people without
>> the economic structure in place to sustain the work.
> 
> Again, we tell what it is and what it is not. I think support is
> loaded term here. Instead we can create lengthy documentation
> explaining to a detail lifecycle and testing certain container had to
> pass before it lands in dockerhub. Maybe add link to particular set of
> jobs that container had passed. Only thing we can offer is automated
> and transparent process of publishing. On top of that? You are on your
> own. But even within these boundaries, a lot of people could have
> better experience of running OpenStack...

That totally makes sense. Supporting builds like "a published container
passed some test scenarios for our CI gates, here is a link to
particular set of jobs that container had passed" benefits all and has
nothing to the production use cases and guarantees nothing in terms of
supporting them.

> 
>> Doug
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-05-17 11:36:40 -0700:
> On 17 May 2017 at 11:04, Doug Hellmann  wrote:
>
> > You've presented some positive scenarios. Here's a worst case
> > situation that I'm worried about.
> >
> > Suppose in a few months the top several companies contributing to
> > kolla decide to pull out of or reduce their contributions to
> > OpenStack.  IBM, Intel, Oracle, and Cisco either lay folks off or
> > redirect their efforts to other projects.  Maybe they start
> > contributing directly to kubernetes. The kolla team is hit badly,
> > and all of the people from that team who know how the container
> > publishing jobs work are gone.
> 
> There are only 2 ways to defend against that: diverse community, which
> we have. If Intel, Red Hat, Oracle, Cisco and IBM back out of
> OpenStack, we'd still have almost 50% of contributors. I think we'll
> much more likely to survive than most of other Big Tent projects. In
> fact, I'd think with our current diversity, that we'll survive for as
> long as OpenStack survives.
> 
> Also all the more reasons why *we shouldn't build images personally*,
> we should have autonomous process to do it for us.
> 
> > The day after everyone says goodbye, the build breaks. Maybe a bad
> > patch lands, or maybe some upstream assumption changes. The issue
> > isn't with the infra jobs themselves. The break means no new container
> > images are being published. Since there's not much of a kolla team
> > any more, it looks like it will be a while before anyone has time
> > to figure out how to fix the problem.
> 
> > Later that same day, a new zero-day exploit is announced in a
> > component included in all or most of those images. Something that
> > isn't developed in the community, such as OpenSSL or glibc. The
> > exploit allows a complete breach of any app running with it. All
> > existing published containers include the bad bits and need to be
> > updated.
> 
> I guess this is problem of all the software ever written. If community
> dies around it, people who uses it are in lots of trouble. One way to
> make sure it won't happen is to get involved yourself to make sure you
> can fix what is broken for you. This is how open source works. In
> Kolla most of our contributors are actually operators who run these
> very containers in their own infrastructure. This is where our
> diversity comes from. We aren't distro and that makes us, and our
> users, more protected from this scenario.
> 
> If nova loses all of it's community, and someone finds critical bug in
> nova that allows hackers to gain access to vm data, there will be
> nobody to fix it, that's bad right? Same argument can be made. We
> aren't discussing deleting Nova tho right?

I think there's a difference there, because of the way nova and the
other components currently have an intermediary doing the distribution.

> > Contrast that with a scenario in which consumers either take
> > responsibility for their systems by building their own images, by
> > collaborating directly with other consumers to share the resources
> > needed to build those images, or by paying a third-party a sustainable
> > amount of money to build images for them. In any of those cases,
> > there is an incentive for the responsible party to be ready and
> > able to produce new images in a timely manner. Consumers of the
> > images know exactly where to go for support when they have problems.
> > Issues in those images don't reflect on the community in any way,
> > because we were not involved in producing them.
> 
> Unless as you said build system breaks, then they are equally screwed
> locally. Unless someone fix it, and they can fix it for openstack
> infra too. Difference is, for OpenStack infra it's whole community
> that can fix it where local it's just you. That's the strength of open
> source.

The difference is that it's definitely not the community's problem in
that case. I'm looking at this from a community perspective, and not the
deployer or operator.

> > As I said at the start of this thread, we've long avoided building
> > and supporting simple operating system style packages of the
> > components we produce. I am still struggling to understand how
> > building more complex artifacts, including bits over which we have
> > little or no control, is somehow more sustainable than those simple
> > packages.
> 
> Binaries are built as standalone projects. Nova-api has no
> dependencies build into .rpm. If issue you just described would happen
> in any of projects openstack uses as dependency, in any of these [1],
> same argument applies. We pin specific versions in upper constraints.
> I'm willing to bet money of it that if today one of these libs would
> release CVE, there is good change we won't find out.

I don't understand what you're saying here. How do constraints we use in
our test gate enter into this?

> Bottom line, yes, we do *package* today with PIP. Exactly same issues
> 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 17 May 2017 at 11:36, Michał Jastrzębski  wrote:
> On 17 May 2017 at 11:04, Doug Hellmann  wrote:
>> Excerpts from Michał Jastrzębski's message of 2017-05-17 07:47:31 -0700:
>>> On 17 May 2017 at 04:14, Chris Dent  wrote:
>>> > On Wed, 17 May 2017, Thierry Carrez wrote:
>>> >
>>> >> Back to container image world, if we refresh those images daily and they
>>> >> are not versioned or archived (basically you can only use the latest and
>>> >> can't really access past dailies), I think we'd be in a similar situation
>>> >> ?
>>> >
>>> >
>>> > Yes, this.
>>>
>>> I think it's not a bad idea to message "you are responsible for
>>> archving your containers". Do that, combine it with good toolset that
>>> helps users determine versions of packages and other metadata and
>>> we'll end up with something that itself would be greatly appreciated.
>>>
>>> Few potential user stories.
>>>
>>> I have OpenStack <100 nodes and need every single one of them, hence
>>> no CI. At the same time I want to have fresh packages to avoid CVEs. I
>>> deploy kolla with tip-of-the-stable-branch and setup cronjob that will
>>> upgrade it every week. Because my scenerio is quite typical and
>>> containers already ran through gates that tests my scenerio, I'm good.
>>>
>>> Another one:
>>>
>>> I have 300+ node cloud, heavy CI and security team examining every
>>> container. While I could build containers locally, downloading them is
>>> just simpler and effectively the same (after all, it's containers
>>> being tested not build process). Every download our security team
>>> scrutinize contaniers and uses toolset Kolla provides to help them.
>>> Additional benefit is that on top of our CI these images went through
>>> Kolla CI which is nice, more testing is always good.
>>>
>>> And another one
>>>
>>> We are Kolla community. We want to provide testing for full release
>>> upgrades every day in gates, to make sure OpenStack and Kolla is
>>> upgradable and improve general user experience of upgrades. Because
>>> infra is resource constrained, we cannot afford building 2 sets of
>>> containers (stable and master) and doing deploy->test->upgrade->test.
>>> However because we have these cached containers, that are fresh and
>>> passed CI for deploy, we can just use them! Now effectively we're not
>>> only testing Kolla's correctness of upgrade procedure but also all the
>>> other project team upgrades! Oh, it seems Nova merged something that
>>> negatively affects upgrades, let's make sure they are aware!
>>>
>>> And last one, which cannot be underestimated
>>>
>>> I am CTO of some company and I've heard OpenStack is no longer hard to
>>> deploy, I'll just download kolla-ansible and try. I'll follow this
>>> guide that deploys simple OpenStack with 2 commands and few small
>>> configs, and it's done! Super simple! We're moving to OpenStack and
>>> start contributing tomorrow!
>>>
>>> Please, let's solve messaging problems, put burden of archiving on
>>> users, whatever it takes to protect our community from wrong
>>> expectations, but not kill this effort. There are very real and
>>> immediate benefits to OpenStack as a whole if we do this.
>>>
>>> Cheers,
>>> Michal
>>
>> You've presented some positive scenarios. Here's a worst case
>> situation that I'm worried about.
>>
>> Suppose in a few months the top several companies contributing to
>> kolla decide to pull out of or reduce their contributions to
>> OpenStack.  IBM, Intel, Oracle, and Cisco either lay folks off or
>> redirect their efforts to other projects.  Maybe they start
>> contributing directly to kubernetes. The kolla team is hit badly,
>> and all of the people from that team who know how the container
>> publishing jobs work are gone.
>
> There are only 2 ways to defend against that: diverse community, which
> we have. If Intel, Red Hat, Oracle, Cisco and IBM back out of
> OpenStack, we'd still have almost 50% of contributors. I think we'll
> much more likely to survive than most of other Big Tent projects. In
> fact, I'd think with our current diversity, that we'll survive for as
> long as OpenStack survives.

Diverse community and off-by-one errors;) I was meaning to say diverse
community and involvement.

>
> Also all the more reasons why *we shouldn't build images personally*,
> we should have autonomous process to do it for us.
>
>> The day after everyone says goodbye, the build breaks. Maybe a bad
>> patch lands, or maybe some upstream assumption changes. The issue
>> isn't with the infra jobs themselves. The break means no new container
>> images are being published. Since there's not much of a kolla team
>> any more, it looks like it will be a while before anyone has time
>> to figure out how to fix the problem.
>
>> Later that same day, a new zero-day exploit is announced in a
>> component included in all or most of those images. Something that
>> isn't developed in the community, such as OpenSSL or 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 17 May 2017 at 11:04, Doug Hellmann  wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-17 07:47:31 -0700:
>> On 17 May 2017 at 04:14, Chris Dent  wrote:
>> > On Wed, 17 May 2017, Thierry Carrez wrote:
>> >
>> >> Back to container image world, if we refresh those images daily and they
>> >> are not versioned or archived (basically you can only use the latest and
>> >> can't really access past dailies), I think we'd be in a similar situation
>> >> ?
>> >
>> >
>> > Yes, this.
>>
>> I think it's not a bad idea to message "you are responsible for
>> archving your containers". Do that, combine it with good toolset that
>> helps users determine versions of packages and other metadata and
>> we'll end up with something that itself would be greatly appreciated.
>>
>> Few potential user stories.
>>
>> I have OpenStack <100 nodes and need every single one of them, hence
>> no CI. At the same time I want to have fresh packages to avoid CVEs. I
>> deploy kolla with tip-of-the-stable-branch and setup cronjob that will
>> upgrade it every week. Because my scenerio is quite typical and
>> containers already ran through gates that tests my scenerio, I'm good.
>>
>> Another one:
>>
>> I have 300+ node cloud, heavy CI and security team examining every
>> container. While I could build containers locally, downloading them is
>> just simpler and effectively the same (after all, it's containers
>> being tested not build process). Every download our security team
>> scrutinize contaniers and uses toolset Kolla provides to help them.
>> Additional benefit is that on top of our CI these images went through
>> Kolla CI which is nice, more testing is always good.
>>
>> And another one
>>
>> We are Kolla community. We want to provide testing for full release
>> upgrades every day in gates, to make sure OpenStack and Kolla is
>> upgradable and improve general user experience of upgrades. Because
>> infra is resource constrained, we cannot afford building 2 sets of
>> containers (stable and master) and doing deploy->test->upgrade->test.
>> However because we have these cached containers, that are fresh and
>> passed CI for deploy, we can just use them! Now effectively we're not
>> only testing Kolla's correctness of upgrade procedure but also all the
>> other project team upgrades! Oh, it seems Nova merged something that
>> negatively affects upgrades, let's make sure they are aware!
>>
>> And last one, which cannot be underestimated
>>
>> I am CTO of some company and I've heard OpenStack is no longer hard to
>> deploy, I'll just download kolla-ansible and try. I'll follow this
>> guide that deploys simple OpenStack with 2 commands and few small
>> configs, and it's done! Super simple! We're moving to OpenStack and
>> start contributing tomorrow!
>>
>> Please, let's solve messaging problems, put burden of archiving on
>> users, whatever it takes to protect our community from wrong
>> expectations, but not kill this effort. There are very real and
>> immediate benefits to OpenStack as a whole if we do this.
>>
>> Cheers,
>> Michal
>
> You've presented some positive scenarios. Here's a worst case
> situation that I'm worried about.
>
> Suppose in a few months the top several companies contributing to
> kolla decide to pull out of or reduce their contributions to
> OpenStack.  IBM, Intel, Oracle, and Cisco either lay folks off or
> redirect their efforts to other projects.  Maybe they start
> contributing directly to kubernetes. The kolla team is hit badly,
> and all of the people from that team who know how the container
> publishing jobs work are gone.

There are only 2 ways to defend against that: diverse community, which
we have. If Intel, Red Hat, Oracle, Cisco and IBM back out of
OpenStack, we'd still have almost 50% of contributors. I think we'll
much more likely to survive than most of other Big Tent projects. In
fact, I'd think with our current diversity, that we'll survive for as
long as OpenStack survives.

Also all the more reasons why *we shouldn't build images personally*,
we should have autonomous process to do it for us.

> The day after everyone says goodbye, the build breaks. Maybe a bad
> patch lands, or maybe some upstream assumption changes. The issue
> isn't with the infra jobs themselves. The break means no new container
> images are being published. Since there's not much of a kolla team
> any more, it looks like it will be a while before anyone has time
> to figure out how to fix the problem.

> Later that same day, a new zero-day exploit is announced in a
> component included in all or most of those images. Something that
> isn't developed in the community, such as OpenSSL or glibc. The
> exploit allows a complete breach of any app running with it. All
> existing published containers include the bad bits and need to be
> updated.

I guess this is problem of all the software ever written. If community
dies around it, people who uses it are in lots of 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-05-17 07:47:31 -0700:
> On 17 May 2017 at 04:14, Chris Dent  wrote:
> > On Wed, 17 May 2017, Thierry Carrez wrote:
> >
> >> Back to container image world, if we refresh those images daily and they
> >> are not versioned or archived (basically you can only use the latest and
> >> can't really access past dailies), I think we'd be in a similar situation
> >> ?
> >
> >
> > Yes, this.
> 
> I think it's not a bad idea to message "you are responsible for
> archving your containers". Do that, combine it with good toolset that
> helps users determine versions of packages and other metadata and
> we'll end up with something that itself would be greatly appreciated.
> 
> Few potential user stories.
> 
> I have OpenStack <100 nodes and need every single one of them, hence
> no CI. At the same time I want to have fresh packages to avoid CVEs. I
> deploy kolla with tip-of-the-stable-branch and setup cronjob that will
> upgrade it every week. Because my scenerio is quite typical and
> containers already ran through gates that tests my scenerio, I'm good.
> 
> Another one:
> 
> I have 300+ node cloud, heavy CI and security team examining every
> container. While I could build containers locally, downloading them is
> just simpler and effectively the same (after all, it's containers
> being tested not build process). Every download our security team
> scrutinize contaniers and uses toolset Kolla provides to help them.
> Additional benefit is that on top of our CI these images went through
> Kolla CI which is nice, more testing is always good.
> 
> And another one
> 
> We are Kolla community. We want to provide testing for full release
> upgrades every day in gates, to make sure OpenStack and Kolla is
> upgradable and improve general user experience of upgrades. Because
> infra is resource constrained, we cannot afford building 2 sets of
> containers (stable and master) and doing deploy->test->upgrade->test.
> However because we have these cached containers, that are fresh and
> passed CI for deploy, we can just use them! Now effectively we're not
> only testing Kolla's correctness of upgrade procedure but also all the
> other project team upgrades! Oh, it seems Nova merged something that
> negatively affects upgrades, let's make sure they are aware!
> 
> And last one, which cannot be underestimated
> 
> I am CTO of some company and I've heard OpenStack is no longer hard to
> deploy, I'll just download kolla-ansible and try. I'll follow this
> guide that deploys simple OpenStack with 2 commands and few small
> configs, and it's done! Super simple! We're moving to OpenStack and
> start contributing tomorrow!
> 
> Please, let's solve messaging problems, put burden of archiving on
> users, whatever it takes to protect our community from wrong
> expectations, but not kill this effort. There are very real and
> immediate benefits to OpenStack as a whole if we do this.
> 
> Cheers,
> Michal

You've presented some positive scenarios. Here's a worst case
situation that I'm worried about.

Suppose in a few months the top several companies contributing to
kolla decide to pull out of or reduce their contributions to
OpenStack.  IBM, Intel, Oracle, and Cisco either lay folks off or
redirect their efforts to other projects.  Maybe they start
contributing directly to kubernetes. The kolla team is hit badly,
and all of the people from that team who know how the container
publishing jobs work are gone.

The day after everyone says goodbye, the build breaks. Maybe a bad
patch lands, or maybe some upstream assumption changes. The issue
isn't with the infra jobs themselves. The break means no new container
images are being published. Since there's not much of a kolla team
any more, it looks like it will be a while before anyone has time
to figure out how to fix the problem.

Later that same day, a new zero-day exploit is announced in a
component included in all or most of those images. Something that
isn't developed in the community, such as OpenSSL or glibc. The
exploit allows a complete breach of any app running with it. All
existing published containers include the bad bits and need to be
updated.

We now have an unknown number of clouds running containers built
by the community with major security holes. The team responsible
for maintaining those images is a shambles, but even if they weren't
the automation isn't working, so no new images can be published.
The consumers of the existing containers haven't bothered to set
up build pipelines of their own, because why bother? Even though
we've clearly said the images "we" publish are for our own testing,
they have found it irresistibly convenient to use them and move on
with their lives.

When the exploit is announced, they start clamoring for new container
images, and become understandably irate when we say we didn't think
they would be using them in production and they *shouldn't have*
and their problems are not 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Fox, Kevin M]

You can do that, but doesn't play well with orchestration systems such as k8s 
as it removes its ability to know when upgraded containers appear.

Thanks,
Kevin

* As always, sorry for top posting, but my organization does not allow me the 
choice of mail software.

From: Doug Hellmann [d...@doughellmann.com]
Sent: Wednesday, May 17, 2017 8:55 AM
To: openstack-dev
Subject: Re: [openstack-dev]
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
do we want to be publishing binary container images?

Excerpts from Chris Dent's message of 2017-05-17 12:14:40 +0100:
> On Wed, 17 May 2017, Thierry Carrez wrote:
>
> > Back to container image world, if we refresh those images daily and they
> > are not versioned or archived (basically you can only use the latest and
> > can't really access past dailies), I think we'd be in a similar situation ?
>
> Yes, this.
>

Is that how container publishing works? Can we overwrite an existing
archive, so that there is only ever 1 version of a published container
at any given time?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Fox, Kevin M]

What kolla's been discussing is having something like:
4.0.0-1, 4.0.0-2, 4.0.0-3, etc.
only keeping the most recent two. and then aliases for:
4.0.0 pointing to the newest.

This allows helm upgrade to atomically roll/forward back properly. If you drop 
releases, k8s can't properly do atomic upgrades. You will get inconsistent 
rollouts and will not know which containers are old and have the security 
issues. Knowing there is a newer -revision number also notifies you that you 
are running something old and need to update.

Thanks,
Kevin

From: Chris Dent [cdent...@anticdent.org]
Sent: Wednesday, May 17, 2017 4:14 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

On Wed, 17 May 2017, Thierry Carrez wrote:

> Back to container image world, if we refresh those images daily and they
> are not versioned or archived (basically you can only use the latest and
> can't really access past dailies), I think we'd be in a similar situation ?

Yes, this.

--
Chris Dent  ┬──┬◡ﾉ(° -°ﾉ)   https://anticdent.org/
freenode: cdent tw: @anticdent
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Steven Dake (stdake)]

Doug,

When a docker image is pushed to dockerhub (or quay.io, etc) a tag is 
specified.  If the tag already exists, it is overwritten.

Regards
-steve

-Original Message-
From: Doug Hellmann <d...@doughellmann.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Date: Wednesday, May 17, 2017 at 8:55 AM
To: openstack-dev <openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev]
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
    do we want to be publishing binary container images?

Excerpts from Chris Dent's message of 2017-05-17 12:14:40 +0100:
> On Wed, 17 May 2017, Thierry Carrez wrote:
> 
> > Back to container image world, if we refresh those images daily and they
> > are not versioned or archived (basically you can only use the latest and
> > can't really access past dailies), I think we'd be in a similar 
situation ?
> 
> Yes, this.
> 

Is that how container publishing works? Can we overwrite an existing
archive, so that there is only ever 1 version of a published container
at any given time?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 17 May 2017 at 08:55, Doug Hellmann  wrote:
> Excerpts from Chris Dent's message of 2017-05-17 12:14:40 +0100:
>> On Wed, 17 May 2017, Thierry Carrez wrote:
>>
>> > Back to container image world, if we refresh those images daily and they
>> > are not versioned or archived (basically you can only use the latest and
>> > can't really access past dailies), I think we'd be in a similar situation ?
>>
>> Yes, this.
>>
>
> Is that how container publishing works? Can we overwrite an existing
> archive, so that there is only ever 1 version of a published container
> at any given time?

We can do it either way, but that's how we want it, top of stable
branch daily + top of master.

> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2017-05-17 12:19:22 +0200:
> Sean Dague wrote:
> > On 05/16/2017 02:39 PM, Doug Hellmann wrote:
> >> Excerpts from Michał Jastrzębski's message of 2017-05-16 09:51:00 -0700:
> >>> One thing I struggle with is...well...how does *not having* built
> >>> containers help with that? If your company have full time security
> >>> team, they can check our containers prior to deployment. If your
> >>> company doesn't, then building locally will be subject to same risks
> >>> as downloading from dockerhub. Difference is, dockerhub containers
> >>> were tested in our CI to extend that our CI allows. No matter whether
> >>> or not you have your own security team, local CI, staging env, that
> >>> will be just a little bit of testing on top of that which you get for
> >>> free, and I think that's value enough for users to push for this.
> >>
> >> The benefit of not building images ourselves is that we are clearly
> >> communicating that the responsibility for maintaining the images
> >> falls on whoever *does* build them. There can be no question in any
> >> user's mind that the community somehow needs to maintain the content
> >> of the images for them, just because we're publishing new images
> >> at some regular cadence.
> > 
> > +1. It is really easy to think that saying "don't use this in
> > production" prevents people from using it in production. See: User
> > Survey 2017 and the number of folks reporting DevStack as their
> > production deployment tool.
> > 
> > We need to not only manage artifacts, but expectations. And with all the
> > confusion of projects in the openstack git namespace being officially
> > blessed openstack projects over the past few years, I can't imagine
> > people not thinking that openstack infra generated content in dockerhub
> > is officially supported content.
> 
> I totally agree, although I think daily rebuilds / per-commit rebuilds,
> together with a properly named repository, might limit expectations
> enough to remove the "supported" part of your sentence.
> 
> As a parallel, we refresh per-commit a Nova master source code tarball
> (nova-master.tar.gz). If a vulnerability is introduced in master but was
> never "released" with a version number, we silently fix it in master (no
> OSSA advisory published). People tracking master are supposed to be
> continuously tracking master.
> 
> Back to container image world, if we refresh those images daily and they
> are not versioned or archived (basically you can only use the latest and
> can't really access past dailies), I think we'd be in a similar situation ?
> 

The source tarballs are not production deployment tools and only
contain code for one project at a time and it is all our code, so
we don't have to track issues in any other components. The same
differences apply to the artifacts we publish to PyPI and NPM. So
it's similar, but different.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Chris Dent's message of 2017-05-17 12:14:40 +0100:
> On Wed, 17 May 2017, Thierry Carrez wrote:
> 
> > Back to container image world, if we refresh those images daily and they
> > are not versioned or archived (basically you can only use the latest and
> > can't really access past dailies), I think we'd be in a similar situation ?
> 
> Yes, this.
> 

Is that how container publishing works? Can we overwrite an existing
archive, so that there is only ever 1 version of a published container
at any given time?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Steven Dake (stdake)]

Michael,

There has been a lot of cost and risk analysis in this thread.

What hasn’t really been discussed at great detail is the “benefit analysis” 
which you have started.  I think we are all clear on the risks and the costs.

If we as a technical community are going to place a line in the sand and state 
“thou shall not ship containers to dockerhub” because of the risks inherent in 
such behavior, we are not integrating properly with the emerging container 
ecosystem.  Expecting operators to build their own images is a viable path 
forward.  Unfortunately, the lack of automation introduces significant 
cognitive load for many (based upon the Q in the #openstack-kolla channel on 
a daily basis).  This cognitive load could be (incorrectly) perceived by many 
to be “OpenStack doesn’t care about integrating with adjacent communities.”

On balance, the benefits to OpenStack of your proposal outweigh the costs.

Regards
-steve


-Original Message-
From: Michał Jastrzębski <inc...@gmail.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Date: Wednesday, May 17, 2017 at 7:47 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

On 17 May 2017 at 04:14, Chris Dent <cdent...@anticdent.org> wrote:
> On Wed, 17 May 2017, Thierry Carrez wrote:
>
>> Back to container image world, if we refresh those images daily and they
>> are not versioned or archived (basically you can only use the latest and
>> can't really access past dailies), I think we'd be in a similar situation
>> ?
>
>
> Yes, this.

I think it's not a bad idea to message "you are responsible for
archving your containers". Do that, combine it with good toolset that
helps users determine versions of packages and other metadata and
we'll end up with something that itself would be greatly appreciated.

Few potential user stories.

I have OpenStack <100 nodes and need every single one of them, hence
no CI. At the same time I want to have fresh packages to avoid CVEs. I
deploy kolla with tip-of-the-stable-branch and setup cronjob that will
upgrade it every week. Because my scenerio is quite typical and
containers already ran through gates that tests my scenerio, I'm good.

Another one:

I have 300+ node cloud, heavy CI and security team examining every
container. While I could build containers locally, downloading them is
just simpler and effectively the same (after all, it's containers
being tested not build process). Every download our security team
scrutinize contaniers and uses toolset Kolla provides to help them.
Additional benefit is that on top of our CI these images went through
Kolla CI which is nice, more testing is always good.

And another one

We are Kolla community. We want to provide testing for full release
upgrades every day in gates, to make sure OpenStack and Kolla is
upgradable and improve general user experience of upgrades. Because
infra is resource constrained, we cannot afford building 2 sets of
containers (stable and master) and doing deploy->test->upgrade->test.
However because we have these cached containers, that are fresh and
passed CI for deploy, we can just use them! Now effectively we're not
only testing Kolla's correctness of upgrade procedure but also all the
other project team upgrades! Oh, it seems Nova merged something that
negatively affects upgrades, let's make sure they are aware!

And last one, which cannot be underestimated

I am CTO of some company and I've heard OpenStack is no longer hard to
deploy, I'll just download kolla-ansible and try. I'll follow this
guide that deploys simple OpenStack with 2 commands and few small
configs, and it's done! Super simple! We're moving to OpenStack and
start contributing tomorrow!

Please, let's solve messaging problems, put burden of archiving on
users, whatever it takes to protect our community from wrong
expectations, but not kill this effort. There are very real and
immediate benefits to OpenStack as a whole if we do this.

Cheers,
Michal

> --
> Chris Dent  ┬──┬◡ﾉ(° -°ﾉ)   https://anticdent.org/
> freenode: cdent tw: @anticdent
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 17 May 2017 at 04:14, Chris Dent  wrote:
> On Wed, 17 May 2017, Thierry Carrez wrote:
>
>> Back to container image world, if we refresh those images daily and they
>> are not versioned or archived (basically you can only use the latest and
>> can't really access past dailies), I think we'd be in a similar situation
>> ?
>
>
> Yes, this.

I think it's not a bad idea to message "you are responsible for
archving your containers". Do that, combine it with good toolset that
helps users determine versions of packages and other metadata and
we'll end up with something that itself would be greatly appreciated.

Few potential user stories.

I have OpenStack <100 nodes and need every single one of them, hence
no CI. At the same time I want to have fresh packages to avoid CVEs. I
deploy kolla with tip-of-the-stable-branch and setup cronjob that will
upgrade it every week. Because my scenerio is quite typical and
containers already ran through gates that tests my scenerio, I'm good.

Another one:

I have 300+ node cloud, heavy CI and security team examining every
container. While I could build containers locally, downloading them is
just simpler and effectively the same (after all, it's containers
being tested not build process). Every download our security team
scrutinize contaniers and uses toolset Kolla provides to help them.
Additional benefit is that on top of our CI these images went through
Kolla CI which is nice, more testing is always good.

And another one

We are Kolla community. We want to provide testing for full release
upgrades every day in gates, to make sure OpenStack and Kolla is
upgradable and improve general user experience of upgrades. Because
infra is resource constrained, we cannot afford building 2 sets of
containers (stable and master) and doing deploy->test->upgrade->test.
However because we have these cached containers, that are fresh and
passed CI for deploy, we can just use them! Now effectively we're not
only testing Kolla's correctness of upgrade procedure but also all the
other project team upgrades! Oh, it seems Nova merged something that
negatively affects upgrades, let's make sure they are aware!

And last one, which cannot be underestimated

I am CTO of some company and I've heard OpenStack is no longer hard to
deploy, I'll just download kolla-ansible and try. I'll follow this
guide that deploys simple OpenStack with 2 commands and few small
configs, and it's done! Super simple! We're moving to OpenStack and
start contributing tomorrow!

Please, let's solve messaging problems, put burden of archiving on
users, whatever it takes to protect our community from wrong
expectations, but not kill this effort. There are very real and
immediate benefits to OpenStack as a whole if we do this.

Cheers,
Michal

> --
> Chris Dent  ┬──┬◡ﾉ(° -°ﾉ)   https://anticdent.org/
> freenode: cdent tw: @anticdent
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Chris Dent]

On Wed, 17 May 2017, Thierry Carrez wrote:


Back to container image world, if we refresh those images daily and they
are not versioned or archived (basically you can only use the latest and
can't really access past dailies), I think we'd be in a similar situation ?


Yes, this.

--
Chris Dent  ┬──┬◡ﾉ(° -°ﾉ)   https://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Thierry Carrez]

Sean Dague wrote:
> On 05/16/2017 02:39 PM, Doug Hellmann wrote:
>> Excerpts from Michał Jastrzębski's message of 2017-05-16 09:51:00 -0700:
>>> One thing I struggle with is...well...how does *not having* built
>>> containers help with that? If your company have full time security
>>> team, they can check our containers prior to deployment. If your
>>> company doesn't, then building locally will be subject to same risks
>>> as downloading from dockerhub. Difference is, dockerhub containers
>>> were tested in our CI to extend that our CI allows. No matter whether
>>> or not you have your own security team, local CI, staging env, that
>>> will be just a little bit of testing on top of that which you get for
>>> free, and I think that's value enough for users to push for this.
>>
>> The benefit of not building images ourselves is that we are clearly
>> communicating that the responsibility for maintaining the images
>> falls on whoever *does* build them. There can be no question in any
>> user's mind that the community somehow needs to maintain the content
>> of the images for them, just because we're publishing new images
>> at some regular cadence.
> 
> +1. It is really easy to think that saying "don't use this in
> production" prevents people from using it in production. See: User
> Survey 2017 and the number of folks reporting DevStack as their
> production deployment tool.
> 
> We need to not only manage artifacts, but expectations. And with all the
> confusion of projects in the openstack git namespace being officially
> blessed openstack projects over the past few years, I can't imagine
> people not thinking that openstack infra generated content in dockerhub
> is officially supported content.

I totally agree, although I think daily rebuilds / per-commit rebuilds,
together with a properly named repository, might limit expectations
enough to remove the "supported" part of your sentence.

As a parallel, we refresh per-commit a Nova master source code tarball
(nova-master.tar.gz). If a vulnerability is introduced in master but was
never "released" with a version number, we silently fix it in master (no
OSSA advisory published). People tracking master are supposed to be
continuously tracking master.

Back to container image world, if we refresh those images daily and they
are not versioned or archived (basically you can only use the latest and
can't really access past dailies), I think we'd be in a similar situation ?

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Jeremy Stanley]

On 2017-05-16 19:56:31 + (+), Fox, Kevin M wrote:
[...]
> Lets provide the tools to make it as easy as possible to identify
> containers with issues, and allow upgrading the system to newer
> ones.
> 
> Which CVE's are on the system is somewhat less important then
> being able to get to newer versions installed easily. Right now,
> thats probably harder then it should be. If its hard, people won't
> do it.
[...]

My point (which I've trimmed because I don't have the patience to
undo your top-posting at the moment) was that security expectations
for these images should be clearly documented and communicated,
that's all. I'm not sure what you were reading into it.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Fox, Kevin M]

Security is a spectrum, not a boolean. I know some sites that have instituted 
super long/complex password requirements. The end result is usually humans just 
writing pw's down on stickies then since its too hard to remember, making 
security worse, not better. Humans are always the weakest link in the security 
system and must be taken into account.

There are people that will do things insecurely. If we can make it much easier 
for them to get access to much more fresh stuff rather then build it themselves 
(which they wont), the community as a whole will be better off. There will be 
far fewer clouds out there with known bad CVE's baked in.

Lets provide the tools to make it as easy as possible to identify containers 
with issues, and allow upgrading the system to newer ones.

Which CVE's are on the system is somewhat less important then being able to get 
to newer versions installed easily. Right now, thats probably harder then it 
should be. If its hard, people won't do it.

Fresh images are just a step in that process.

Thanks,
Kevin

From: Jeremy Stanley [fu...@yuggoth.org]
Sent: Tuesday, May 16, 2017 12:36 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

On 2017-05-16 11:46:14 -0700 (-0700), Michał Jastrzębski wrote:
[...]
> So CVE tracking might not be required by us. Since we still use
> distro packages under the hood, we can just use these.
[...]

I think the question is how I, as a semi-clueful downstream user of
your images, can tell whether the image I'm deploying has fixes for
some specific recently disclosed vulnerability. It sounds like your
answer is that I should compare the package manifest against the
versions listed on the distro's CVE tracker or similar service? That
should be prominently documented, perhaps in a highly visible FAQ
list.

> Since we'd rebuild daily, that alone would ensure timely update to
> our containers. What we can promise to potential users is that
> containers out there were built lately (24hrs)
[...]

As outlined elsewhere in the thread, there are a myriad of reasons
why this could end up not being the case from time to time so I can
only assume your definition of "promise" differs from mine (and
unfortunately, from most people who might be trying to decide
whether it's safe to rely on these images in a sensitive/production
environment).
--
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 12:36, Jeremy Stanley  wrote:
> On 2017-05-16 11:46:14 -0700 (-0700), Michał Jastrzębski wrote:
> [...]
>> So CVE tracking might not be required by us. Since we still use
>> distro packages under the hood, we can just use these.
> [...]
>
> I think the question is how I, as a semi-clueful downstream user of
> your images, can tell whether the image I'm deploying has fixes for
> some specific recently disclosed vulnerability. It sounds like your
> answer is that I should compare the package manifest against the
> versions listed on the distro's CVE tracker or similar service? That
> should be prominently documented, perhaps in a highly visible FAQ
> list.

One thing we've been working on prior to summit was manifesto of
versions - I think we can provide single file with all the versions of
packages in container, we can add track of CI jobs that led containers
to this place, all the informations that semi-careful downstream user
can use to help him/her to determine what's that they're getting. I'm
all for that kind of features.

>> Since we'd rebuild daily, that alone would ensure timely update to
>> our containers. What we can promise to potential users is that
>> containers out there were built lately (24hrs)
> [...]
>
> As outlined elsewhere in the thread, there are a myriad of reasons
> why this could end up not being the case from time to time so I can
> only assume your definition of "promise" differs from mine (and
> unfortunately, from most people who might be trying to decide
> whether it's safe to rely on these images in a sensitive/production
> environment).

By "promise" I mean clear documentation of where containers came from
and what did they pass. After that, take it or leave it.

> --
> Jeremy Stanley
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Jeremy Stanley]

On 2017-05-16 11:46:14 -0700 (-0700), Michał Jastrzębski wrote:
[...]
> So CVE tracking might not be required by us. Since we still use
> distro packages under the hood, we can just use these.
[...]

I think the question is how I, as a semi-clueful downstream user of
your images, can tell whether the image I'm deploying has fixes for
some specific recently disclosed vulnerability. It sounds like your
answer is that I should compare the package manifest against the
versions listed on the distro's CVE tracker or similar service? That
should be prominently documented, perhaps in a highly visible FAQ
list.

> Since we'd rebuild daily, that alone would ensure timely update to
> our containers. What we can promise to potential users is that
> containers out there were built lately (24hrs)
[...]

As outlined elsewhere in the thread, there are a myriad of reasons
why this could end up not being the case from time to time so I can
only assume your definition of "promise" differs from mine (and
unfortunately, from most people who might be trying to decide
whether it's safe to rely on these images in a sensitive/production
environment).
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Fox, Kevin M]

We can put warnings all over it and if folks choose to ignore them, then its 
they who took the risk and get to keep the pieces when it breaks. Some folks 
are crazy enough to run devstack in production. But does that mean we should 
just abandon devstack? No. of course not. I don't think we should hold back 
OpenStack from the benefits this provides just because a few folks might 
possibly do something bad with it. We do what we can to allow folks to 
recognize bad ideas. But if they want to do it anyway, thats the freedom of 
open source. They can. They get to deal with the fallout though, not us. If we 
always catered to the lowest common denominator we would never get anywhere.

Thanks,
Kevin

From: Doug Hellmann [d...@doughellmann.com]
Sent: Tuesday, May 16, 2017 11:49 AM
To: openstack-dev
Subject: Re: [openstack-dev]
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
do we want to be publishing binary container images?

Excerpts from Michał Jastrzębski's message of 2017-05-16 11:38:19 -0700:
> On 16 May 2017 at 11:27, Doug Hellmann <d...@doughellmann.com> wrote:
> > Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700:
> >> So another consideration. Do you think whole rule of "not building
> >> binares" should be reconsidered? We are kind of new use case here. We
> >> aren't distro but we are packagers (kind of). I don't think putting us
> >> on equal footing as Red Hat, Canonical or other companies is correct
> >> here.
> >>
> >> K8s is something we want to work with, and what we are discussing is
> >> central to how k8s is used. K8s community creates this culture of
> >> "organic packages" built by anyone, most of companies/projects already
> >> have semi-official container images and I think expectations on
> >> quality of these are well...none? You get what you're given and if you
> >> don't agree, there is always way to reproduce this yourself.
> >>
> >> [Another huge snip]
> >>
> >
> > I wanted to have the discussion, but my position for now is that
> > we should continue as we have been and not change the policy.
> >
> > I don't have a problem with any individual or group of individuals
> > publishing their own organic packages. The issue I have is with
> > making sure it is clear those *are* "organic" and not officially
> > supported by the broader community. One way to do that is to say
> > they need to be built somewhere other than on our shared infrastructure.
> > There may be other ways, though, so I'm looking for input on that.
>
> What I was trying to say here is, current discussion aside, maybe we
> should revise this "not supported by broader community" rule. They may
> very well be supported to a certain point. Support is not just yes or
> no, it's all the levels in between. I think we can afford *some* level
> of official support, even if that some level means best effort made by
> community. If Kolla community, not an individual like myself, would
> like to support these images best to our ability, why aren't we
> allowed? As long as we are crystal clear what is scope of our support,
> why can't we do it? I think we've already proven that it's going to be
> tremendously useful for a lot of people, even in a shape we discuss
> today, that is "best effort, you still need to validate it for
> yourself"...

Right, I understood that. So far I haven't heard anything to change
my mind, though.

I think you're underestimating the amount of risk you're taking on
for yourselves and by extension the rest of the community, and
introducing to potential consumers of the images, by promising to
support production deployments with a small team of people without
the economic structure in place to sustain the work.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Fox, Kevin M]

And bandwidth can be conserved by only uploading images that actually changed 
in non trivial ways (packages were updated, not just logfile with a new 
timestamp)

Thanks,
Keivn

From: Michał Jastrzębski [inc...@gmail.com]
Sent: Tuesday, May 16, 2017 11:46 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

On 16 May 2017 at 11:33, Doug Hellmann <d...@doughellmann.com> wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-16 08:20:17 -0700:
>> On 16 May 2017 at 08:12, Doug Hellmann <d...@doughellmann.com> wrote:
>> > Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700:
>> >> On 16 May 2017 at 06:20, Flavio Percoco <fla...@redhat.com> wrote:
>> >> > On 16/05/17 14:08 +0200, Thierry Carrez wrote:
>> >> >>
>> >> >> Flavio Percoco wrote:
>> >> >>>
>> >> >>> From a release perspective, as Doug mentioned, we've avoided releasing
>> >> >>> projects
>> >> >>> in any kind of built form. This was also one of the concerns I raised
>> >> >>> when
>> >> >>> working on the proposal to support other programming languages. The
>> >> >>> problem of
>> >> >>> releasing built images goes beyond the infrastructure requirements. 
>> >> >>> It's
>> >> >>> the
>> >> >>> message and the guarantees implied with the built product itself that 
>> >> >>> are
>> >> >>> the
>> >> >>> concern here. And I tend to agree with Doug that this might be a 
>> >> >>> problem
>> >> >>> for us
>> >> >>> as a community. Unfortunately, putting your name, Michal, as contact
>> >> >>> point is
>> >> >>> not enough. Kolla is not the only project producing container images 
>> >> >>> and
>> >> >>> we need
>> >> >>> to be consistent in the way we release these images.
>> >> >>>
>> >> >>> Nothing prevents people for building their own images and uploading 
>> >> >>> them
>> >> >>> to
>> >> >>> dockerhub. Having this as part of the OpenStack's pipeline is a 
>> >> >>> problem.
>> >> >>
>> >> >>
>> >> >> I totally subscribe to the concerns around publishing binaries (under
>> >> >> any form), and the expectations in terms of security maintenance that 
>> >> >> it
>> >> >> would set on the publisher. At the same time, we need to have images
>> >> >> available, for convenience and testing. So what is the best way to
>> >> >> achieve that without setting strong security maintenance expectations
>> >> >> for the OpenStack community ? We have several options:
>> >> >>
>> >> >> 1/ Have third-parties publish images
>> >> >> It is the current situation. The issue is that the Kolla team (and
>> >> >> likely others) would rather automate the process and use OpenStack
>> >> >> infrastructure for it.
>> >> >>
>> >> >> 2/ Have third-parties publish images, but through OpenStack infra
>> >> >> This would allow to automate the process, but it would be a bit weird 
>> >> >> to
>> >> >> use common infra resources to publish in a private repo.
>> >> >>
>> >> >> 3/ Publish transient (per-commit or daily) images
>> >> >> A "daily build" (especially if you replace it every day) would set
>> >> >> relatively-limited expectations in terms of maintenance. It would end 
>> >> >> up
>> >> >> picking up security updates in upstream layers, even if not 
>> >> >> immediately.
>> >> >>
>> >> >> 4/ Publish images and own them
>> >> >> Staff release / VMT / stable team in a way that lets us properly own
>> >> >> those images and publish them officially.
>> >> >>
>> >> >> Personally I think (4) is not realistic. I think we could make (3) 
>> >> >> work,
>> >>

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 11:49, Doug Hellmann  wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-16 11:38:19 -0700:
>> On 16 May 2017 at 11:27, Doug Hellmann  wrote:
>> > Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700:
>> >> So another consideration. Do you think whole rule of "not building
>> >> binares" should be reconsidered? We are kind of new use case here. We
>> >> aren't distro but we are packagers (kind of). I don't think putting us
>> >> on equal footing as Red Hat, Canonical or other companies is correct
>> >> here.
>> >>
>> >> K8s is something we want to work with, and what we are discussing is
>> >> central to how k8s is used. K8s community creates this culture of
>> >> "organic packages" built by anyone, most of companies/projects already
>> >> have semi-official container images and I think expectations on
>> >> quality of these are well...none? You get what you're given and if you
>> >> don't agree, there is always way to reproduce this yourself.
>> >>
>> >> [Another huge snip]
>> >>
>> >
>> > I wanted to have the discussion, but my position for now is that
>> > we should continue as we have been and not change the policy.
>> >
>> > I don't have a problem with any individual or group of individuals
>> > publishing their own organic packages. The issue I have is with
>> > making sure it is clear those *are* "organic" and not officially
>> > supported by the broader community. One way to do that is to say
>> > they need to be built somewhere other than on our shared infrastructure.
>> > There may be other ways, though, so I'm looking for input on that.
>>
>> What I was trying to say here is, current discussion aside, maybe we
>> should revise this "not supported by broader community" rule. They may
>> very well be supported to a certain point. Support is not just yes or
>> no, it's all the levels in between. I think we can afford *some* level
>> of official support, even if that some level means best effort made by
>> community. If Kolla community, not an individual like myself, would
>> like to support these images best to our ability, why aren't we
>> allowed? As long as we are crystal clear what is scope of our support,
>> why can't we do it? I think we've already proven that it's going to be
>> tremendously useful for a lot of people, even in a shape we discuss
>> today, that is "best effort, you still need to validate it for
>> yourself"...
>
> Right, I understood that. So far I haven't heard anything to change
> my mind, though.
>
> I think you're underestimating the amount of risk you're taking on
> for yourselves and by extension the rest of the community, and
> introducing to potential consumers of the images, by promising to
> support production deployments with a small team of people without
> the economic structure in place to sustain the work.

Again, we tell what it is and what it is not. I think support is
loaded term here. Instead we can create lengthy documentation
explaining to a detail lifecycle and testing certain container had to
pass before it lands in dockerhub. Maybe add link to particular set of
jobs that container had passed. Only thing we can offer is automated
and transparent process of publishing. On top of that? You are on your
own. But even within these boundaries, a lot of people could have
better experience of running OpenStack...

> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Sean Dague]

On 05/16/2017 02:39 PM, Doug Hellmann wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-16 09:51:00 -0700:
>> On 16 May 2017 at 09:40, Clint Byrum  wrote:
>>>
>>> What's at stake isn't so much "how do we get the bits to the users" but
>>> "how do we only get bits to users that they need". If you build and push
>>> daily, do you expect all of your users to also _pull_ daily? Redeploy
>>> all their containers? How do you detect that there's new CVE-fixing
>>> stuff in a daily build?
>>>
>>> This is really the realm of distributors that have full-time security
>>> teams tracking issues and providing support to paying customers.
>>>
>>> So I think this is a fine idea, however, it needs to include a commitment
>>> for a full-time paid security team who weighs in on every change to
>>> the manifest. Otherwise we're just lobbing time bombs into our users'
>>> data-centers.
>>
>> One thing I struggle with is...well...how does *not having* built
>> containers help with that? If your company have full time security
>> team, they can check our containers prior to deployment. If your
>> company doesn't, then building locally will be subject to same risks
>> as downloading from dockerhub. Difference is, dockerhub containers
>> were tested in our CI to extend that our CI allows. No matter whether
>> or not you have your own security team, local CI, staging env, that
>> will be just a little bit of testing on top of that which you get for
>> free, and I think that's value enough for users to push for this.
> 
> The benefit of not building images ourselves is that we are clearly
> communicating that the responsibility for maintaining the images
> falls on whoever *does* build them. There can be no question in any
> user's mind that the community somehow needs to maintain the content
> of the images for them, just because we're publishing new images
> at some regular cadence.

+1. It is really easy to think that saying "don't use this in
production" prevents people from using it in production. See: User
Survey 2017 and the number of folks reporting DevStack as their
production deployment tool.

We need to not only manage artifacts, but expectations. And with all the
confusion of projects in the openstack git namespace being officially
blessed openstack projects over the past few years, I can't imagine
people not thinking that openstack infra generated content in dockerhub
is officially supported content.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-05-16 11:38:19 -0700:
> On 16 May 2017 at 11:27, Doug Hellmann  wrote:
> > Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700:
> >> So another consideration. Do you think whole rule of "not building
> >> binares" should be reconsidered? We are kind of new use case here. We
> >> aren't distro but we are packagers (kind of). I don't think putting us
> >> on equal footing as Red Hat, Canonical or other companies is correct
> >> here.
> >>
> >> K8s is something we want to work with, and what we are discussing is
> >> central to how k8s is used. K8s community creates this culture of
> >> "organic packages" built by anyone, most of companies/projects already
> >> have semi-official container images and I think expectations on
> >> quality of these are well...none? You get what you're given and if you
> >> don't agree, there is always way to reproduce this yourself.
> >>
> >> [Another huge snip]
> >>
> >
> > I wanted to have the discussion, but my position for now is that
> > we should continue as we have been and not change the policy.
> >
> > I don't have a problem with any individual or group of individuals
> > publishing their own organic packages. The issue I have is with
> > making sure it is clear those *are* "organic" and not officially
> > supported by the broader community. One way to do that is to say
> > they need to be built somewhere other than on our shared infrastructure.
> > There may be other ways, though, so I'm looking for input on that.
> 
> What I was trying to say here is, current discussion aside, maybe we
> should revise this "not supported by broader community" rule. They may
> very well be supported to a certain point. Support is not just yes or
> no, it's all the levels in between. I think we can afford *some* level
> of official support, even if that some level means best effort made by
> community. If Kolla community, not an individual like myself, would
> like to support these images best to our ability, why aren't we
> allowed? As long as we are crystal clear what is scope of our support,
> why can't we do it? I think we've already proven that it's going to be
> tremendously useful for a lot of people, even in a shape we discuss
> today, that is "best effort, you still need to validate it for
> yourself"...

Right, I understood that. So far I haven't heard anything to change
my mind, though.

I think you're underestimating the amount of risk you're taking on
for yourselves and by extension the rest of the community, and
introducing to potential consumers of the images, by promising to
support production deployments with a small team of people without
the economic structure in place to sustain the work.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 11:33, Doug Hellmann  wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-16 08:20:17 -0700:
>> On 16 May 2017 at 08:12, Doug Hellmann  wrote:
>> > Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700:
>> >> On 16 May 2017 at 06:20, Flavio Percoco  wrote:
>> >> > On 16/05/17 14:08 +0200, Thierry Carrez wrote:
>> >> >>
>> >> >> Flavio Percoco wrote:
>> >> >>>
>> >> >>> From a release perspective, as Doug mentioned, we've avoided releasing
>> >> >>> projects
>> >> >>> in any kind of built form. This was also one of the concerns I raised
>> >> >>> when
>> >> >>> working on the proposal to support other programming languages. The
>> >> >>> problem of
>> >> >>> releasing built images goes beyond the infrastructure requirements. 
>> >> >>> It's
>> >> >>> the
>> >> >>> message and the guarantees implied with the built product itself that 
>> >> >>> are
>> >> >>> the
>> >> >>> concern here. And I tend to agree with Doug that this might be a 
>> >> >>> problem
>> >> >>> for us
>> >> >>> as a community. Unfortunately, putting your name, Michal, as contact
>> >> >>> point is
>> >> >>> not enough. Kolla is not the only project producing container images 
>> >> >>> and
>> >> >>> we need
>> >> >>> to be consistent in the way we release these images.
>> >> >>>
>> >> >>> Nothing prevents people for building their own images and uploading 
>> >> >>> them
>> >> >>> to
>> >> >>> dockerhub. Having this as part of the OpenStack's pipeline is a 
>> >> >>> problem.
>> >> >>
>> >> >>
>> >> >> I totally subscribe to the concerns around publishing binaries (under
>> >> >> any form), and the expectations in terms of security maintenance that 
>> >> >> it
>> >> >> would set on the publisher. At the same time, we need to have images
>> >> >> available, for convenience and testing. So what is the best way to
>> >> >> achieve that without setting strong security maintenance expectations
>> >> >> for the OpenStack community ? We have several options:
>> >> >>
>> >> >> 1/ Have third-parties publish images
>> >> >> It is the current situation. The issue is that the Kolla team (and
>> >> >> likely others) would rather automate the process and use OpenStack
>> >> >> infrastructure for it.
>> >> >>
>> >> >> 2/ Have third-parties publish images, but through OpenStack infra
>> >> >> This would allow to automate the process, but it would be a bit weird 
>> >> >> to
>> >> >> use common infra resources to publish in a private repo.
>> >> >>
>> >> >> 3/ Publish transient (per-commit or daily) images
>> >> >> A "daily build" (especially if you replace it every day) would set
>> >> >> relatively-limited expectations in terms of maintenance. It would end 
>> >> >> up
>> >> >> picking up security updates in upstream layers, even if not 
>> >> >> immediately.
>> >> >>
>> >> >> 4/ Publish images and own them
>> >> >> Staff release / VMT / stable team in a way that lets us properly own
>> >> >> those images and publish them officially.
>> >> >>
>> >> >> Personally I think (4) is not realistic. I think we could make (3) 
>> >> >> work,
>> >> >> and I prefer it to (2). If all else fails, we should keep (1).
>> >> >
>> >> >
>> >> > Agreed #4 is a bit unrealistic.
>> >> >
>> >> > Not sure I understand the difference between #2 and #3. Is it just the
>> >> > cadence?
>> >> >
>> >> > I'd prefer for these builds to have a daily cadence because it sets the
>> >> > expectations w.r.t maintenance right: "These images are daily builds 
>> >> > and not
>> >> > certified releases. For stable builds you're better off building it
>> >> > yourself"
>> >>
>> >> And daily builds are exactly what I wanted in the first place:) We
>> >> probably will keep publishing release packages too, but we can be so
>> >> called 3rd party. I also agree [4] is completely unrealistic and I
>> >> would be against putting such heavy burden of responsibility on any
>> >> community, including Kolla.
>> >>
>> >> While daily cadence will send message that it's not stable, truth will
>> >> be that it will be more stable than what people would normally build
>> >> locally (again, it passes more gates), but I'm totally fine in not
>> >> saying that and let people decide how they want to use it.
>> >>
>> >> So, can we move on with implementation?
>> >
>> > I don't want the images published to docker hub. Are they still useful
>> > to you if they aren't published?
>>
>> What do you mean? We need images available...whether it's dockerhub,
>> infra-hosted registry or any other way to have them, we need to be
>> able to have images that are available and fresh without building.
>> Dockerhub/quay.io is least problems for infra team/resources.
>
> There are 2 separate concerns.
>
> The first concern is whether this is a good idea at all, from a
> policy perspective. Do we have the people to maintain the images,
> track CVEs, etc.? Do we have the response time to update or remove
> bad images? 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Joshua Harlow]

My guess is same with octavia.

https://github.com/openstack/octavia/tree/master/diskimage-create#diskimage-builder-script-for-creating-octavia-amphora-images

-Josh

Fox, Kevin M wrote:

+1. ironic and trove have the same issues as well. lowering the bar in order to 
kick the tires will help OpenStack a lot in adoption.

From: Sean Dague [s...@dague.net]
Sent: Tuesday, May 16, 2017 6:28 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

On 05/16/2017 09:24 AM, Doug Hellmann wrote:




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-05-16 09:51:00 -0700:
> On 16 May 2017 at 09:40, Clint Byrum  wrote:
> >
> > What's at stake isn't so much "how do we get the bits to the users" but
> > "how do we only get bits to users that they need". If you build and push
> > daily, do you expect all of your users to also _pull_ daily? Redeploy
> > all their containers? How do you detect that there's new CVE-fixing
> > stuff in a daily build?
> >
> > This is really the realm of distributors that have full-time security
> > teams tracking issues and providing support to paying customers.
> >
> > So I think this is a fine idea, however, it needs to include a commitment
> > for a full-time paid security team who weighs in on every change to
> > the manifest. Otherwise we're just lobbing time bombs into our users'
> > data-centers.
> 
> One thing I struggle with is...well...how does *not having* built
> containers help with that? If your company have full time security
> team, they can check our containers prior to deployment. If your
> company doesn't, then building locally will be subject to same risks
> as downloading from dockerhub. Difference is, dockerhub containers
> were tested in our CI to extend that our CI allows. No matter whether
> or not you have your own security team, local CI, staging env, that
> will be just a little bit of testing on top of that which you get for
> free, and I think that's value enough for users to push for this.

The benefit of not building images ourselves is that we are clearly
communicating that the responsibility for maintaining the images
falls on whoever *does* build them. There can be no question in any
user's mind that the community somehow needs to maintain the content
of the images for them, just because we're publishing new images
at some regular cadence.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Fox, Kevin M]

I'm not sure I follow the problem. The containers being built don't pull from 
infra's mirrors, so they can be secure containers. Once built, during 
deploy/testing, they don't install anything, so shouldn't have any issues there 
either.

Am I misunderstanding?

Thanks,
Kevin


From: Sam Yaple [sam...@yaple.net]
Sent: Tuesday, May 16, 2017 7:11 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

I would like to bring up a subject that hasn't really been discussed in this 
thread yet, forgive me if I missed an email mentioning this.

What I personally would like to see is a publishing infrastructure to allow 
pushing built images to an internal infra mirror/repo/registry for consumption 
of internal infra jobs (deployment tools like kolla-ansible and 
openstack-ansible). The images built from infra mirrors with security turned 
off are perfect for testing internally to infra.

If you build images properly in infra, then you will have an image that is not 
security checked (no gpg verification of packages) and completely unverifiable. 
These are absolutely not images we want to push to DockerHub/quay for obvious 
reasons. Security and verification being chief among them. They are absolutely 
not images that should ever be run in production and are only suited for 
testing. These are the only types of images that can come out of infra.

Thanks,
SamYaple

On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski 
<inc...@gmail.com<mailto:inc...@gmail.com>> wrote:
On 16 May 2017 at 06:22, Doug Hellmann 
<d...@doughellmann.com<mailto:d...@doughellmann.com>> wrote:
> Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
>> Flavio Percoco wrote:
>> > From a release perspective, as Doug mentioned, we've avoided releasing 
>> > projects
>> > in any kind of built form. This was also one of the concerns I raised when
>> > working on the proposal to support other programming languages. The 
>> > problem of
>> > releasing built images goes beyond the infrastructure requirements. It's 
>> > the
>> > message and the guarantees implied with the built product itself that are 
>> > the
>> > concern here. And I tend to agree with Doug that this might be a problem 
>> > for us
>> > as a community. Unfortunately, putting your name, Michal, as contact point 
>> > is
>> > not enough. Kolla is not the only project producing container images and 
>> > we need
>> > to be consistent in the way we release these images.
>> >
>> > Nothing prevents people for building their own images and uploading them to
>> > dockerhub. Having this as part of the OpenStack's pipeline is a problem.
>>
>> I totally subscribe to the concerns around publishing binaries (under
>> any form), and the expectations in terms of security maintenance that it
>> would set on the publisher. At the same time, we need to have images
>> available, for convenience and testing. So what is the best way to
>> achieve that without setting strong security maintenance expectations
>> for the OpenStack community ? We have several options:
>>
>> 1/ Have third-parties publish images
>> It is the current situation. The issue is that the Kolla team (and
>> likely others) would rather automate the process and use OpenStack
>> infrastructure for it.
>>
>> 2/ Have third-parties publish images, but through OpenStack infra
>> This would allow to automate the process, but it would be a bit weird to
>> use common infra resources to publish in a private repo.
>>
>> 3/ Publish transient (per-commit or daily) images
>> A "daily build" (especially if you replace it every day) would set
>> relatively-limited expectations in terms of maintenance. It would end up
>> picking up security updates in upstream layers, even if not immediately.
>>
>> 4/ Publish images and own them
>> Staff release / VMT / stable team in a way that lets us properly own
>> those images and publish them officially.
>>
>> Personally I think (4) is not realistic. I think we could make (3) work,
>> and I prefer it to (2). If all else fails, we should keep (1).
>>
>
> At the forum we talked about putting test images on a "private"
> repository hosted on openstack.org<http://openstack.org> somewhere. I think 
> that's option
> 3 from your list?
>
> Paul may be able to shed more light on the details of the technology
> (maybe it's just an Apache-served repo, rather than a full blo

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 11:27, Doug Hellmann  wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700:
>> So another consideration. Do you think whole rule of "not building
>> binares" should be reconsidered? We are kind of new use case here. We
>> aren't distro but we are packagers (kind of). I don't think putting us
>> on equal footing as Red Hat, Canonical or other companies is correct
>> here.
>>
>> K8s is something we want to work with, and what we are discussing is
>> central to how k8s is used. K8s community creates this culture of
>> "organic packages" built by anyone, most of companies/projects already
>> have semi-official container images and I think expectations on
>> quality of these are well...none? You get what you're given and if you
>> don't agree, there is always way to reproduce this yourself.
>>
>> [Another huge snip]
>>
>
> I wanted to have the discussion, but my position for now is that
> we should continue as we have been and not change the policy.
>
> I don't have a problem with any individual or group of individuals
> publishing their own organic packages. The issue I have is with
> making sure it is clear those *are* "organic" and not officially
> supported by the broader community. One way to do that is to say
> they need to be built somewhere other than on our shared infrastructure.
> There may be other ways, though, so I'm looking for input on that.

What I was trying to say here is, current discussion aside, maybe we
should revise this "not supported by broader community" rule. They may
very well be supported to a certain point. Support is not just yes or
no, it's all the levels in between. I think we can afford *some* level
of official support, even if that some level means best effort made by
community. If Kolla community, not an individual like myself, would
like to support these images best to our ability, why aren't we
allowed? As long as we are crystal clear what is scope of our support,
why can't we do it? I think we've already proven that it's going to be
tremendously useful for a lot of people, even in a shape we discuss
today, that is "best effort, you still need to validate it for
yourself"...

> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-05-16 08:20:17 -0700:
> On 16 May 2017 at 08:12, Doug Hellmann  wrote:
> > Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700:
> >> On 16 May 2017 at 06:20, Flavio Percoco  wrote:
> >> > On 16/05/17 14:08 +0200, Thierry Carrez wrote:
> >> >>
> >> >> Flavio Percoco wrote:
> >> >>>
> >> >>> From a release perspective, as Doug mentioned, we've avoided releasing
> >> >>> projects
> >> >>> in any kind of built form. This was also one of the concerns I raised
> >> >>> when
> >> >>> working on the proposal to support other programming languages. The
> >> >>> problem of
> >> >>> releasing built images goes beyond the infrastructure requirements. 
> >> >>> It's
> >> >>> the
> >> >>> message and the guarantees implied with the built product itself that 
> >> >>> are
> >> >>> the
> >> >>> concern here. And I tend to agree with Doug that this might be a 
> >> >>> problem
> >> >>> for us
> >> >>> as a community. Unfortunately, putting your name, Michal, as contact
> >> >>> point is
> >> >>> not enough. Kolla is not the only project producing container images 
> >> >>> and
> >> >>> we need
> >> >>> to be consistent in the way we release these images.
> >> >>>
> >> >>> Nothing prevents people for building their own images and uploading 
> >> >>> them
> >> >>> to
> >> >>> dockerhub. Having this as part of the OpenStack's pipeline is a 
> >> >>> problem.
> >> >>
> >> >>
> >> >> I totally subscribe to the concerns around publishing binaries (under
> >> >> any form), and the expectations in terms of security maintenance that it
> >> >> would set on the publisher. At the same time, we need to have images
> >> >> available, for convenience and testing. So what is the best way to
> >> >> achieve that without setting strong security maintenance expectations
> >> >> for the OpenStack community ? We have several options:
> >> >>
> >> >> 1/ Have third-parties publish images
> >> >> It is the current situation. The issue is that the Kolla team (and
> >> >> likely others) would rather automate the process and use OpenStack
> >> >> infrastructure for it.
> >> >>
> >> >> 2/ Have third-parties publish images, but through OpenStack infra
> >> >> This would allow to automate the process, but it would be a bit weird to
> >> >> use common infra resources to publish in a private repo.
> >> >>
> >> >> 3/ Publish transient (per-commit or daily) images
> >> >> A "daily build" (especially if you replace it every day) would set
> >> >> relatively-limited expectations in terms of maintenance. It would end up
> >> >> picking up security updates in upstream layers, even if not immediately.
> >> >>
> >> >> 4/ Publish images and own them
> >> >> Staff release / VMT / stable team in a way that lets us properly own
> >> >> those images and publish them officially.
> >> >>
> >> >> Personally I think (4) is not realistic. I think we could make (3) work,
> >> >> and I prefer it to (2). If all else fails, we should keep (1).
> >> >
> >> >
> >> > Agreed #4 is a bit unrealistic.
> >> >
> >> > Not sure I understand the difference between #2 and #3. Is it just the
> >> > cadence?
> >> >
> >> > I'd prefer for these builds to have a daily cadence because it sets the
> >> > expectations w.r.t maintenance right: "These images are daily builds and 
> >> > not
> >> > certified releases. For stable builds you're better off building it
> >> > yourself"
> >>
> >> And daily builds are exactly what I wanted in the first place:) We
> >> probably will keep publishing release packages too, but we can be so
> >> called 3rd party. I also agree [4] is completely unrealistic and I
> >> would be against putting such heavy burden of responsibility on any
> >> community, including Kolla.
> >>
> >> While daily cadence will send message that it's not stable, truth will
> >> be that it will be more stable than what people would normally build
> >> locally (again, it passes more gates), but I'm totally fine in not
> >> saying that and let people decide how they want to use it.
> >>
> >> So, can we move on with implementation?
> >
> > I don't want the images published to docker hub. Are they still useful
> > to you if they aren't published?
> 
> What do you mean? We need images available...whether it's dockerhub,
> infra-hosted registry or any other way to have them, we need to be
> able to have images that are available and fresh without building.
> Dockerhub/quay.io is least problems for infra team/resources.

There are 2 separate concerns.

The first concern is whether this is a good idea at all, from a
policy perspective. Do we have the people to maintain the images,
track CVEs, etc.? Do we have the response time to update or remove
bad images? Can we, as a community, actually staff the support to
an appropriate level? Or, can we clearly communicate that we do not
support the images for production use and effectively avoid having
someone start to rely on them?

The 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Jeremy Stanley's message of 2017-05-16 17:41:28 +:
> On 2017-05-16 11:17:31 -0400 (-0400), Doug Hellmann wrote:
> > Excerpts from Sam Yaple's message of 2017-05-16 14:11:18 +:
> [...]
> > > If you build images properly in infra, then you will have an image that is
> > > not security checked (no gpg verification of packages) and completely
> > > unverifiable. These are absolutely not images we want to push to
> > > DockerHub/quay for obvious reasons. Security and verification being chief
> > > among them. They are absolutely not images that should ever be run in
> > > production and are only suited for testing. These are the only types of
> > > images that can come out of infra.
> > 
> > This sounds like an implementation detail of option 3? I think not
> > signing the images does help indicate that they're not meant to be used
> > in production environments.
> [...]
> 
> I'm pretty sure Sam wasn't talking about whether or not the images
> which get built are signed, but whether or not the package manager
> used when building the images vets the distro packages it retrieves
> (the Ubuntu package mirror we maintain in our CI doesn't have
> "secure APT" signatures available for its indices so we disable that
> security measure by default in the CI system to allow us to use
> those mirrors). Point being, if images are built in the upstream CI
> with packages from our Ubuntu package mirror then they are (at least
> at present) not suitable for production use from a security
> perspective for this particular reason even in absence of the other
> concerns expressed.

Thanks for clarifying; that makes more sense.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-05-16 09:46:19 -0700:
> So another consideration. Do you think whole rule of "not building
> binares" should be reconsidered? We are kind of new use case here. We
> aren't distro but we are packagers (kind of). I don't think putting us
> on equal footing as Red Hat, Canonical or other companies is correct
> here.
> 
> K8s is something we want to work with, and what we are discussing is
> central to how k8s is used. K8s community creates this culture of
> "organic packages" built by anyone, most of companies/projects already
> have semi-official container images and I think expectations on
> quality of these are well...none? You get what you're given and if you
> don't agree, there is always way to reproduce this yourself.
> 
> [Another huge snip]
> 

I wanted to have the discussion, but my position for now is that
we should continue as we have been and not change the policy.

I don't have a problem with any individual or group of individuals
publishing their own organic packages. The issue I have is with
making sure it is clear those *are* "organic" and not officially
supported by the broader community. One way to do that is to say
they need to be built somewhere other than on our shared infrastructure.
There may be other ways, though, so I'm looking for input on that.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Fox, Kevin M]

+1. ironic and trove have the same issues as well. lowering the bar in order to 
kick the tires will help OpenStack a lot in adoption.

From: Sean Dague [s...@dague.net]
Sent: Tuesday, May 16, 2017 6:28 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

On 05/16/2017 09:24 AM, Doug Hellmann wrote:
> Excerpts from Luigi Toscano's message of 2017-05-16 11:50:53 +0200:
>> On Monday, 15 May 2017 21:12:16 CEST Doug Hellmann wrote:
>>> Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700:
>>>
>>>> On 15 May 2017 at 10:34, Doug Hellmann <d...@doughellmann.com> wrote:
>>>>> I'm raising the issue here to get some more input into how to
>>>>> proceed. Do other people think this concern is overblown? Can we
>>>>> mitigate the risk by communicating through metadata for the images?
>>>>> Should we stick to publishing build instructions (Dockerfiles, or
>>>>> whatever) instead of binary images? Are there other options I haven't
>>>>> mentioned?
>>>>
>>>> Today we do publish build instructions, that's what Kolla is. We also
>>>> publish built containers already, just we do it manually on release
>>>> today. If we decide to block it, I assume we should stop doing that
>>>> too? That will hurt users who uses this piece of Kolla, and I'd hate
>>>> to hurt our users:(
>>>
>>> Well, that's the question. Today we have teams publishing those
>>> images themselves, right? And the proposal is to have infra do it?
>>> That change could be construed to imply that there is more of a
>>> relationship with the images and the rest of the community (remember,
>>> folks outside of the main community activities do not always make
>>> the same distinctions we do about teams). So, before we go ahead
>>> with that, I want to make sure that we all have a chance to discuss
>>> the policy change and its implications.
>>
>> Sorry for hijacking the thread, but we have a similar scenario for example in
>> Sahara. It is about full VM images containing Hadoop/Spark/other_big_data
>> stuff, and not containers, but it's looks really the same.
>> So far ready-made images have been published under 
>> http://sahara-files.mirantis.com/images/upstream/, but we are looking to 
>> have them hosted on
>> openstack.org, just like other artifacts.
>>
>> We asked about this few days ago on openstack-infra@, but no answer so far
>> (the Summit didn't help):
>>
>> http://lists.openstack.org/pipermail/openstack-infra/2017-April/005312.html
>>
>> I think that the answer to the question raised in this thread is definitely
>> going to be relevant for our use case.
>>
>> Ciao
>
> Thanks for raising this. I think the same concerns apply to VM images.

Agreed.

-Sean

--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Fox, Kevin M]

Sorry, I'm a bit late to the discussion.

Over the time I've maintained openstack clouds, I've seen many times, 
integration issues pop up between distro packages and openstack packages. The 
released openstack packages are usually fine. Changes in the distros break 
kolla builds over time. 

In the past Kolla's dealt with this by having users show up, say "the *stuff* 
doesn't build" and then someone on irc scrambles to figure out why it broke. 
This is not a good place to be in. We even suffer through it ourselves with the 
gates. An upstream releases something, everything breaks, and then we spend all 
our time debugging the same problem rather then some of us debugging that and 
the rest making forward progress.

So, we're starting to break up the gate jobs into periodic gates that test 
known good kolla stuff against newer upstream stuff to see if it works. if it 
does, it caches it for the regular gate tests for new patch sets. If it fails, 
then someone has time to look at it without the rest of the gates being broken. 
As part of this process we also produce known working, up to date, openstack 
stable release containers, since we need them for upgrade gate testing.

So, basically to have proper gates in kolla, we have to do all the work of 
building up to date stable/trunk containers that are tested with the gate suite 
of tests.

So, the real question is, can we go the last 2 feet and push them to the docker 
hub rather then doing it manually like we do today and they tend to rot there. 
There is a fair amount of benefit to some users and very little additional cost 
to openstack. I see very little reason not to.

We should still recommend users build it themselves. But for many use cases, 
such as testing the waters, an operator might just want the easiest way to see 
the thing work (pull updated containers from the hub), prove out that its worth 
doing it with Kolla, then either building their own containers for production, 
or better yet, paying a Distro for support.

We want to make it as easy as possible to try out OpenStack. This is one of the 
biggest/most reported problems with OpenStack. Saying, step 1, you must always 
build all the containers yourself is not a part of solving that problem.

Thanks,
Kevin


From: Flavio Percoco [fla...@redhat.com]
Sent: Tuesday, May 16, 2017 6:20 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

On 16/05/17 14:08 +0200, Thierry Carrez wrote:
>Flavio Percoco wrote:
>> From a release perspective, as Doug mentioned, we've avoided releasing 
>> projects
>> in any kind of built form. This was also one of the concerns I raised when
>> working on the proposal to support other programming languages. The problem 
>> of
>> releasing built images goes beyond the infrastructure requirements. It's the
>> message and the guarantees implied with the built product itself that are the
>> concern here. And I tend to agree with Doug that this might be a problem for 
>> us
>> as a community. Unfortunately, putting your name, Michal, as contact point is
>> not enough. Kolla is not the only project producing container images and we 
>> need
>> to be consistent in the way we release these images.
>>
>> Nothing prevents people for building their own images and uploading them to
>> dockerhub. Having this as part of the OpenStack's pipeline is a problem.
>
>I totally subscribe to the concerns around publishing binaries (under
>any form), and the expectations in terms of security maintenance that it
>would set on the publisher. At the same time, we need to have images
>available, for convenience and testing. So what is the best way to
>achieve that without setting strong security maintenance expectations
>for the OpenStack community ? We have several options:
>
>1/ Have third-parties publish images
>It is the current situation. The issue is that the Kolla team (and
>likely others) would rather automate the process and use OpenStack
>infrastructure for it.
>
>2/ Have third-parties publish images, but through OpenStack infra
>This would allow to automate the process, but it would be a bit weird to
>use common infra resources to publish in a private repo.
>
>3/ Publish transient (per-commit or daily) images
>A "daily build" (especially if you replace it every day) would set
>relatively-limited expectations in terms of maintenance. It would end up
>picking up security updates in upstream layers, even if not immediately.
>
>4/ Publish images and own them
>Staff release / VMT / stable team in a way that lets us properly own
>those images and publish them officially.
&

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 10:41, Jeremy Stanley  wrote:
> On 2017-05-16 11:17:31 -0400 (-0400), Doug Hellmann wrote:
>> Excerpts from Sam Yaple's message of 2017-05-16 14:11:18 +:
> [...]
>> > If you build images properly in infra, then you will have an image that is
>> > not security checked (no gpg verification of packages) and completely
>> > unverifiable. These are absolutely not images we want to push to
>> > DockerHub/quay for obvious reasons. Security and verification being chief
>> > among them. They are absolutely not images that should ever be run in
>> > production and are only suited for testing. These are the only types of
>> > images that can come out of infra.
>>
>> This sounds like an implementation detail of option 3? I think not
>> signing the images does help indicate that they're not meant to be used
>> in production environments.
> [...]
>
> I'm pretty sure Sam wasn't talking about whether or not the images
> which get built are signed, but whether or not the package manager
> used when building the images vets the distro packages it retrieves
> (the Ubuntu package mirror we maintain in our CI doesn't have
> "secure APT" signatures available for its indices so we disable that
> security measure by default in the CI system to allow us to use
> those mirrors). Point being, if images are built in the upstream CI
> with packages from our Ubuntu package mirror then they are (at least
> at present) not suitable for production use from a security
> perspective for this particular reason even in absence of the other
> concerns expressed.
> --
> Jeremy Stanley

This is valid concern, but also particularly easy to solve. If we
decide to use nightly builds (or midday in Hawaii? Any timezone with
least traffic would do), we can skip infra mirrors. In fact, that
approach would help us in a different sense as well. Since these
wouldn't be bound to any particular patchset, we could test it to an
extreme, so voting gates for both kolla-ansible and kolla-kubernetes
deployment. I was reluctant to have deploy gates voting inside Kolla,
but that would allow us to do it. In fact, net uplink consumption from
infra would go down, as we won't need to publish tarballs of registry
every commit, we'll do it once a day in a most convenient hour.

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Jeremy Stanley]

On 2017-05-16 11:17:31 -0400 (-0400), Doug Hellmann wrote:
> Excerpts from Sam Yaple's message of 2017-05-16 14:11:18 +:
[...]
> > If you build images properly in infra, then you will have an image that is
> > not security checked (no gpg verification of packages) and completely
> > unverifiable. These are absolutely not images we want to push to
> > DockerHub/quay for obvious reasons. Security and verification being chief
> > among them. They are absolutely not images that should ever be run in
> > production and are only suited for testing. These are the only types of
> > images that can come out of infra.
> 
> This sounds like an implementation detail of option 3? I think not
> signing the images does help indicate that they're not meant to be used
> in production environments.
[...]

I'm pretty sure Sam wasn't talking about whether or not the images
which get built are signed, but whether or not the package manager
used when building the images vets the distro packages it retrieves
(the Ubuntu package mirror we maintain in our CI doesn't have
"secure APT" signatures available for its indices so we disable that
security measure by default in the CI system to allow us to use
those mirrors). Point being, if images are built in the upstream CI
with packages from our Ubuntu package mirror then they are (at least
at present) not suitable for production use from a security
perspective for this particular reason even in absence of the other
concerns expressed.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 09:40, Clint Byrum  wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700:
>> > Container images introduce some extra complexity, over the basic
>> > operating system style packages mentioned above. Due to the way
>> > they are constructed, they are likely to include content we don't
>> > produce ourselves (either in the form of base layers or via including
>> > build tools or other things needed when assembling the full image).
>> > That extra content means there would need to be more tracking of
>> > upstream issues (bugs, CVEs, etc.) to ensure the images are updated
>> > as needed.
>>
>> We can do this by building daily, which was the plan in fact. If we
>> build every day you have at most 24hrs old packages, CVEs and things
>> like that on non-openstack packages are still maintained by distro
>> maintainers.
>>
>
> What's at stake isn't so much "how do we get the bits to the users" but
> "how do we only get bits to users that they need". If you build and push
> daily, do you expect all of your users to also _pull_ daily? Redeploy
> all their containers? How do you detect that there's new CVE-fixing
> stuff in a daily build?
>
> This is really the realm of distributors that have full-time security
> teams tracking issues and providing support to paying customers.
>
> So I think this is a fine idea, however, it needs to include a commitment
> for a full-time paid security team who weighs in on every change to
> the manifest. Otherwise we're just lobbing time bombs into our users'
> data-centers.

One thing I struggle with is...well...how does *not having* built
containers help with that? If your company have full time security
team, they can check our containers prior to deployment. If your
company doesn't, then building locally will be subject to same risks
as downloading from dockerhub. Difference is, dockerhub containers
were tested in our CI to extend that our CI allows. No matter whether
or not you have your own security team, local CI, staging env, that
will be just a little bit of testing on top of that which you get for
free, and I think that's value enough for users to push for this.

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

So another consideration. Do you think whole rule of "not building
binares" should be reconsidered? We are kind of new use case here. We
aren't distro but we are packagers (kind of). I don't think putting us
on equal footing as Red Hat, Canonical or other companies is correct
here.

K8s is something we want to work with, and what we are discussing is
central to how k8s is used. K8s community creates this culture of
"organic packages" built by anyone, most of companies/projects already
have semi-official container images and I think expectations on
quality of these are well...none? You get what you're given and if you
don't agree, there is always way to reproduce this yourself.

[Another huge snip]

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Clint Byrum]

Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700:
> > Container images introduce some extra complexity, over the basic
> > operating system style packages mentioned above. Due to the way
> > they are constructed, they are likely to include content we don't
> > produce ourselves (either in the form of base layers or via including
> > build tools or other things needed when assembling the full image).
> > That extra content means there would need to be more tracking of
> > upstream issues (bugs, CVEs, etc.) to ensure the images are updated
> > as needed.
> 
> We can do this by building daily, which was the plan in fact. If we
> build every day you have at most 24hrs old packages, CVEs and things
> like that on non-openstack packages are still maintained by distro
> maintainers.
> 

What's at stake isn't so much "how do we get the bits to the users" but
"how do we only get bits to users that they need". If you build and push
daily, do you expect all of your users to also _pull_ daily? Redeploy
all their containers? How do you detect that there's new CVE-fixing
stuff in a daily build?

This is really the realm of distributors that have full-time security
teams tracking issues and providing support to paying customers.

So I think this is a fine idea, however, it needs to include a commitment
for a full-time paid security team who weighs in on every change to
the manifest. Otherwise we're just lobbing time bombs into our users'
data-centers.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 08:30, Emilien Macchi  wrote:
> On Tue, May 16, 2017 at 11:12 AM, Doug Hellmann  wrote:
>> Excerpts from Flavio Percoco's message of 2017-05-16 10:07:52 -0400:
>>> On 16/05/17 09:45 -0400, Doug Hellmann wrote:
>>> >Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400:
>>> >> On 15/05/17 11:49 -0700, Michał Jastrzębski wrote:
>>> >> >On 15 May 2017 at 11:19, Davanum Srinivas  wrote:
>>> >> >> Sorry for the top post, Michal, Can you please clarify a couple of 
>>> >> >> things:
>>> >> >>
>>> >> >> 1) Can folks install just one or two services for their specific 
>>> >> >> scenario?
>>> >> >
>>> >> >Yes, that's more of a kolla-ansible feature and require a little bit
>>> >> >of ansible know-how, but entirely possible. Kolla-k8s is built to
>>> >> >allow maximum flexibility in that space.
>>> >> >
>>> >> >> 2) Can the container images from kolla be run on bare docker daemon?
>>> >> >
>>> >> >Yes, but they need to either override our default CMD (kolla_start) or
>>> >> >provide ENVs requred by it, not a huge deal
>>> >> >
>>> >> >> 3) Can someone take the kolla container images from say dockerhub and
>>> >> >> use it without the Kolla framework?
>>> >> >
>>> >> >Yes, there is no such thing as kolla framework really. Our images
>>> >> >follow stable ABI and they can be deployed by any deploy mechanism
>>> >> >that will follow it. We have several users who wrote their own deploy
>>> >> >mechanism from scratch.
>>> >> >
>>> >> >Containers are just blobs with binaries in it. Little things that we
>>> >> >add are kolla_start script to allow our config file management and
>>> >> >some custom startup scripts for things like mariadb to help with
>>> >> >bootstrapping, both are entirely optional.
>>> >>
>>> >> Just as a bonus example, TripleO is currently using kolla images. They 
>>> >> used to
>>> >> be vanilla and they are not anymore but only because TripleO depends on 
>>> >> puppet
>>> >> being in the image, which has nothing to do with kolla.
>>> >>
>>> >> Flavio
>>> >>
>>> >
>>> >When you say "using kolla images," what do you mean? In upstream
>>> >CI tests? On contributors' dev/test systems? Production deployments?
>>>
>>> All of them. Note that TripleO now builds its own "kolla images" (it uses 
>>> the
>>> kolla Dockerfiles and kolla-build) because the dependency of puppet. When I
>>> said, TripleO uses kolla images was intended to answer Dims question on 
>>> whether
>>> these images (or Dockerfiles) can be consumed by other projects.
>>>
>>> Flavio
>>>
>>
>> Ah, OK. So TripleO is using the build instructions for kolla images, but
>> not the binary images being produced today?
>
> Exactly. We have to add Puppet packaging into the list of things we
> want in the binary, that's why we don't consume the binary directly.

And frankly, if we get this thing agreed on, I don't see why TripleO
couldn't publish their images too. If we build technical infra in
Kolla, everyone else can benefit from it.

>> Doug
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Emilien Macchi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Emilien Macchi]

On Tue, May 16, 2017 at 11:12 AM, Doug Hellmann  wrote:
> Excerpts from Flavio Percoco's message of 2017-05-16 10:07:52 -0400:
>> On 16/05/17 09:45 -0400, Doug Hellmann wrote:
>> >Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400:
>> >> On 15/05/17 11:49 -0700, Michał Jastrzębski wrote:
>> >> >On 15 May 2017 at 11:19, Davanum Srinivas  wrote:
>> >> >> Sorry for the top post, Michal, Can you please clarify a couple of 
>> >> >> things:
>> >> >>
>> >> >> 1) Can folks install just one or two services for their specific 
>> >> >> scenario?
>> >> >
>> >> >Yes, that's more of a kolla-ansible feature and require a little bit
>> >> >of ansible know-how, but entirely possible. Kolla-k8s is built to
>> >> >allow maximum flexibility in that space.
>> >> >
>> >> >> 2) Can the container images from kolla be run on bare docker daemon?
>> >> >
>> >> >Yes, but they need to either override our default CMD (kolla_start) or
>> >> >provide ENVs requred by it, not a huge deal
>> >> >
>> >> >> 3) Can someone take the kolla container images from say dockerhub and
>> >> >> use it without the Kolla framework?
>> >> >
>> >> >Yes, there is no such thing as kolla framework really. Our images
>> >> >follow stable ABI and they can be deployed by any deploy mechanism
>> >> >that will follow it. We have several users who wrote their own deploy
>> >> >mechanism from scratch.
>> >> >
>> >> >Containers are just blobs with binaries in it. Little things that we
>> >> >add are kolla_start script to allow our config file management and
>> >> >some custom startup scripts for things like mariadb to help with
>> >> >bootstrapping, both are entirely optional.
>> >>
>> >> Just as a bonus example, TripleO is currently using kolla images. They 
>> >> used to
>> >> be vanilla and they are not anymore but only because TripleO depends on 
>> >> puppet
>> >> being in the image, which has nothing to do with kolla.
>> >>
>> >> Flavio
>> >>
>> >
>> >When you say "using kolla images," what do you mean? In upstream
>> >CI tests? On contributors' dev/test systems? Production deployments?
>>
>> All of them. Note that TripleO now builds its own "kolla images" (it uses the
>> kolla Dockerfiles and kolla-build) because the dependency of puppet. When I
>> said, TripleO uses kolla images was intended to answer Dims question on 
>> whether
>> these images (or Dockerfiles) can be consumed by other projects.
>>
>> Flavio
>>
>
> Ah, OK. So TripleO is using the build instructions for kolla images, but
> not the binary images being produced today?

Exactly. We have to add Puppet packaging into the list of things we
want in the binary, that's why we don't consume the binary directly.

> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Sean McGinnis]

On Tue, May 16, 2017 at 02:08:07PM +0200, Thierry Carrez wrote:
> 
> I totally subscribe to the concerns around publishing binaries (under
> any form), and the expectations in terms of security maintenance that it
> would set on the publisher. At the same time, we need to have images
> available, for convenience and testing. So what is the best way to
> achieve that without setting strong security maintenance expectations
> for the OpenStack community ? We have several options:
> 
> 1/ Have third-parties publish images
> It is the current situation. The issue is that the Kolla team (and
> likely others) would rather automate the process and use OpenStack
> infrastructure for it.
> 
> 2/ Have third-parties publish images, but through OpenStack infra
> This would allow to automate the process, but it would be a bit weird to
> use common infra resources to publish in a private repo.
> 
> 3/ Publish transient (per-commit or daily) images
> A "daily build" (especially if you replace it every day) would set
> relatively-limited expectations in terms of maintenance. It would end up
> picking up security updates in upstream layers, even if not immediately.
> 

I share the concerns around implying support for any of these. But I
also think they could be incredibly useful, and if we don't do it,
there is even more of a chance of multiple "bad" images being published
by others.

I agree having an automated daily image published should give a
reasonable expectation that there is not long term maintenance for
these.

> 4/ Publish images and own them
> Staff release / VMT / stable team in a way that lets us properly own
> those images and publish them officially.
> 
> Personally I think (4) is not realistic. I think we could make (3) work,
> and I prefer it to (2). If all else fails, we should keep (1).
> 
> -- 
> Thierry Carrez (ttx)
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 08:12, Doug Hellmann  wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700:
>> On 16 May 2017 at 06:20, Flavio Percoco  wrote:
>> > On 16/05/17 14:08 +0200, Thierry Carrez wrote:
>> >>
>> >> Flavio Percoco wrote:
>> >>>
>> >>> From a release perspective, as Doug mentioned, we've avoided releasing
>> >>> projects
>> >>> in any kind of built form. This was also one of the concerns I raised
>> >>> when
>> >>> working on the proposal to support other programming languages. The
>> >>> problem of
>> >>> releasing built images goes beyond the infrastructure requirements. It's
>> >>> the
>> >>> message and the guarantees implied with the built product itself that are
>> >>> the
>> >>> concern here. And I tend to agree with Doug that this might be a problem
>> >>> for us
>> >>> as a community. Unfortunately, putting your name, Michal, as contact
>> >>> point is
>> >>> not enough. Kolla is not the only project producing container images and
>> >>> we need
>> >>> to be consistent in the way we release these images.
>> >>>
>> >>> Nothing prevents people for building their own images and uploading them
>> >>> to
>> >>> dockerhub. Having this as part of the OpenStack's pipeline is a problem.
>> >>
>> >>
>> >> I totally subscribe to the concerns around publishing binaries (under
>> >> any form), and the expectations in terms of security maintenance that it
>> >> would set on the publisher. At the same time, we need to have images
>> >> available, for convenience and testing. So what is the best way to
>> >> achieve that without setting strong security maintenance expectations
>> >> for the OpenStack community ? We have several options:
>> >>
>> >> 1/ Have third-parties publish images
>> >> It is the current situation. The issue is that the Kolla team (and
>> >> likely others) would rather automate the process and use OpenStack
>> >> infrastructure for it.
>> >>
>> >> 2/ Have third-parties publish images, but through OpenStack infra
>> >> This would allow to automate the process, but it would be a bit weird to
>> >> use common infra resources to publish in a private repo.
>> >>
>> >> 3/ Publish transient (per-commit or daily) images
>> >> A "daily build" (especially if you replace it every day) would set
>> >> relatively-limited expectations in terms of maintenance. It would end up
>> >> picking up security updates in upstream layers, even if not immediately.
>> >>
>> >> 4/ Publish images and own them
>> >> Staff release / VMT / stable team in a way that lets us properly own
>> >> those images and publish them officially.
>> >>
>> >> Personally I think (4) is not realistic. I think we could make (3) work,
>> >> and I prefer it to (2). If all else fails, we should keep (1).
>> >
>> >
>> > Agreed #4 is a bit unrealistic.
>> >
>> > Not sure I understand the difference between #2 and #3. Is it just the
>> > cadence?
>> >
>> > I'd prefer for these builds to have a daily cadence because it sets the
>> > expectations w.r.t maintenance right: "These images are daily builds and 
>> > not
>> > certified releases. For stable builds you're better off building it
>> > yourself"
>>
>> And daily builds are exactly what I wanted in the first place:) We
>> probably will keep publishing release packages too, but we can be so
>> called 3rd party. I also agree [4] is completely unrealistic and I
>> would be against putting such heavy burden of responsibility on any
>> community, including Kolla.
>>
>> While daily cadence will send message that it's not stable, truth will
>> be that it will be more stable than what people would normally build
>> locally (again, it passes more gates), but I'm totally fine in not
>> saying that and let people decide how they want to use it.
>>
>> So, can we move on with implementation?
>
> I don't want the images published to docker hub. Are they still useful
> to you if they aren't published?

What do you mean? We need images available...whether it's dockerhub,
infra-hosted registry or any other way to have them, we need to be
able to have images that are available and fresh without building.
Dockerhub/quay.io is least problems for infra team/resources.

> Doug
>
>>
>> Thanks!
>> Michal
>>
>> >
>> > Flavio
>> >
>> > --
>> > @flaper87
>> > Flavio Percoco
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Sam Yaple's message of 2017-05-16 14:11:18 +:
> I would like to bring up a subject that hasn't really been discussed in
> this thread yet, forgive me if I missed an email mentioning this.
> 
> What I personally would like to see is a publishing infrastructure to allow
> pushing built images to an internal infra mirror/repo/registry for
> consumption of internal infra jobs (deployment tools like kolla-ansible and
> openstack-ansible). The images built from infra mirrors with security
> turned off are perfect for testing internally to infra.
> 
> If you build images properly in infra, then you will have an image that is
> not security checked (no gpg verification of packages) and completely
> unverifiable. These are absolutely not images we want to push to
> DockerHub/quay for obvious reasons. Security and verification being chief
> among them. They are absolutely not images that should ever be run in
> production and are only suited for testing. These are the only types of
> images that can come out of infra.
> 
> Thanks,
> SamYaple

This sounds like an implementation detail of option 3? I think not
signing the images does help indicate that they're not meant to be used
in production environments.

Is some sort of self-hosted solution a reasonable compromise between
building images in test jobs (which I understand makes them take
extra time) and publishing images to public registries (which is
the thing I object to)?

If self-hosting is reasonable, then we can work out which tool to
use to do it as a second question.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-05-16 06:52:12 -0700:
> On 16 May 2017 at 06:20, Flavio Percoco  wrote:
> > On 16/05/17 14:08 +0200, Thierry Carrez wrote:
> >>
> >> Flavio Percoco wrote:
> >>>
> >>> From a release perspective, as Doug mentioned, we've avoided releasing
> >>> projects
> >>> in any kind of built form. This was also one of the concerns I raised
> >>> when
> >>> working on the proposal to support other programming languages. The
> >>> problem of
> >>> releasing built images goes beyond the infrastructure requirements. It's
> >>> the
> >>> message and the guarantees implied with the built product itself that are
> >>> the
> >>> concern here. And I tend to agree with Doug that this might be a problem
> >>> for us
> >>> as a community. Unfortunately, putting your name, Michal, as contact
> >>> point is
> >>> not enough. Kolla is not the only project producing container images and
> >>> we need
> >>> to be consistent in the way we release these images.
> >>>
> >>> Nothing prevents people for building their own images and uploading them
> >>> to
> >>> dockerhub. Having this as part of the OpenStack's pipeline is a problem.
> >>
> >>
> >> I totally subscribe to the concerns around publishing binaries (under
> >> any form), and the expectations in terms of security maintenance that it
> >> would set on the publisher. At the same time, we need to have images
> >> available, for convenience and testing. So what is the best way to
> >> achieve that without setting strong security maintenance expectations
> >> for the OpenStack community ? We have several options:
> >>
> >> 1/ Have third-parties publish images
> >> It is the current situation. The issue is that the Kolla team (and
> >> likely others) would rather automate the process and use OpenStack
> >> infrastructure for it.
> >>
> >> 2/ Have third-parties publish images, but through OpenStack infra
> >> This would allow to automate the process, but it would be a bit weird to
> >> use common infra resources to publish in a private repo.
> >>
> >> 3/ Publish transient (per-commit or daily) images
> >> A "daily build" (especially if you replace it every day) would set
> >> relatively-limited expectations in terms of maintenance. It would end up
> >> picking up security updates in upstream layers, even if not immediately.
> >>
> >> 4/ Publish images and own them
> >> Staff release / VMT / stable team in a way that lets us properly own
> >> those images and publish them officially.
> >>
> >> Personally I think (4) is not realistic. I think we could make (3) work,
> >> and I prefer it to (2). If all else fails, we should keep (1).
> >
> >
> > Agreed #4 is a bit unrealistic.
> >
> > Not sure I understand the difference between #2 and #3. Is it just the
> > cadence?
> >
> > I'd prefer for these builds to have a daily cadence because it sets the
> > expectations w.r.t maintenance right: "These images are daily builds and not
> > certified releases. For stable builds you're better off building it
> > yourself"
> 
> And daily builds are exactly what I wanted in the first place:) We
> probably will keep publishing release packages too, but we can be so
> called 3rd party. I also agree [4] is completely unrealistic and I
> would be against putting such heavy burden of responsibility on any
> community, including Kolla.
> 
> While daily cadence will send message that it's not stable, truth will
> be that it will be more stable than what people would normally build
> locally (again, it passes more gates), but I'm totally fine in not
> saying that and let people decide how they want to use it.
> 
> So, can we move on with implementation?

I don't want the images published to docker hub. Are they still useful
to you if they aren't published?

Doug

> 
> Thanks!
> Michal
> 
> >
> > Flavio
> >
> > --
> > @flaper87
> > Flavio Percoco
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2017-05-16 10:07:52 -0400:
> On 16/05/17 09:45 -0400, Doug Hellmann wrote:
> >Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400:
> >> On 15/05/17 11:49 -0700, Michał Jastrzębski wrote:
> >> >On 15 May 2017 at 11:19, Davanum Srinivas  wrote:
> >> >> Sorry for the top post, Michal, Can you please clarify a couple of 
> >> >> things:
> >> >>
> >> >> 1) Can folks install just one or two services for their specific 
> >> >> scenario?
> >> >
> >> >Yes, that's more of a kolla-ansible feature and require a little bit
> >> >of ansible know-how, but entirely possible. Kolla-k8s is built to
> >> >allow maximum flexibility in that space.
> >> >
> >> >> 2) Can the container images from kolla be run on bare docker daemon?
> >> >
> >> >Yes, but they need to either override our default CMD (kolla_start) or
> >> >provide ENVs requred by it, not a huge deal
> >> >
> >> >> 3) Can someone take the kolla container images from say dockerhub and
> >> >> use it without the Kolla framework?
> >> >
> >> >Yes, there is no such thing as kolla framework really. Our images
> >> >follow stable ABI and they can be deployed by any deploy mechanism
> >> >that will follow it. We have several users who wrote their own deploy
> >> >mechanism from scratch.
> >> >
> >> >Containers are just blobs with binaries in it. Little things that we
> >> >add are kolla_start script to allow our config file management and
> >> >some custom startup scripts for things like mariadb to help with
> >> >bootstrapping, both are entirely optional.
> >>
> >> Just as a bonus example, TripleO is currently using kolla images. They 
> >> used to
> >> be vanilla and they are not anymore but only because TripleO depends on 
> >> puppet
> >> being in the image, which has nothing to do with kolla.
> >>
> >> Flavio
> >>
> >
> >When you say "using kolla images," what do you mean? In upstream
> >CI tests? On contributors' dev/test systems? Production deployments?
> 
> All of them. Note that TripleO now builds its own "kolla images" (it uses the
> kolla Dockerfiles and kolla-build) because the dependency of puppet. When I
> said, TripleO uses kolla images was intended to answer Dims question on 
> whether
> these images (or Dockerfiles) can be consumed by other projects.
> 
> Flavio
> 

Ah, OK. So TripleO is using the build instructions for kolla images, but
not the binary images being produced today?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Thierry Carrez]

Michał Jastrzębski wrote:
> On 16 May 2017 at 06:20, Flavio Percoco  wrote:
>> I'd prefer for these builds to have a daily cadence because it sets the
>> expectations w.r.t maintenance right: "These images are daily builds and not
>> certified releases. For stable builds you're better off building it
>> yourself"
> 
> And daily builds are exactly what I wanted in the first place:) We
> probably will keep publishing release packages too, but we can be so
> called 3rd party. I also agree [4] is completely unrealistic and I
> would be against putting such heavy burden of responsibility on any
> community, including Kolla.
> 
> While daily cadence will send message that it's not stable, truth will
> be that it will be more stable than what people would normally build
> locally (again, it passes more gates), but I'm totally fine in not
> saying that and let people decide how they want to use it.
> 
> So, can we move on with implementation?

I'm just listing options to help frame the discussion. I still think we
need a global answer on this (for container images and VMs) so I think
it would be great to have a clear TC resolution (picking one of those
options) before moving on with implementation.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Thierry Carrez]

Flavio Percoco wrote:
> On 16/05/17 14:08 +0200, Thierry Carrez wrote:
>> 1/ Have third-parties publish images
>> It is the current situation. The issue is that the Kolla team (and
>> likely others) would rather automate the process and use OpenStack
>> infrastructure for it.
>>
>> 2/ Have third-parties publish images, but through OpenStack infra
>> This would allow to automate the process, but it would be a bit weird to
>> use common infra resources to publish in a private repo.
>>
>> 3/ Publish transient (per-commit or daily) images
>> A "daily build" (especially if you replace it every day) would set
>> relatively-limited expectations in terms of maintenance. It would end up
>> picking up security updates in upstream layers, even if not immediately.
>>
>> 4/ Publish images and own them
>> Staff release / VMT / stable team in a way that lets us properly own
>> those images and publish them officially.
>>
>> Personally I think (4) is not realistic. I think we could make (3) work,
>> and I prefer it to (2). If all else fails, we should keep (1).
> 
> Agreed #4 is a bit unrealistic.
> 
> Not sure I understand the difference between #2 and #3. Is it just the
> cadence?

In #3 the infrastructure ends up publishing to an official
"openstack-daily" repository. In #2 the infrastructure job ends up
publishing to some "flavios-garage" repository.

-- 
Thierry Carrez (ttx)



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 07:11, Sam Yaple  wrote:
> I would like to bring up a subject that hasn't really been discussed in this
> thread yet, forgive me if I missed an email mentioning this.
>
> What I personally would like to see is a publishing infrastructure to allow
> pushing built images to an internal infra mirror/repo/registry for
> consumption of internal infra jobs (deployment tools like kolla-ansible and
> openstack-ansible). The images built from infra mirrors with security turned
> off are perfect for testing internally to infra.
>
> If you build images properly in infra, then you will have an image that is
> not security checked (no gpg verification of packages) and completely
> unverifiable. These are absolutely not images we want to push to
> DockerHub/quay for obvious reasons. Security and verification being chief
> among them. They are absolutely not images that should ever be run in
> production and are only suited for testing. These are the only types of
> images that can come out of infra.

So I guess we need new feature:) since we can test gpg packages...

> Thanks,
> SamYaple
>
> On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski 
> wrote:
>>
>> On 16 May 2017 at 06:22, Doug Hellmann  wrote:
>> > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
>> >> Flavio Percoco wrote:
>> >> > From a release perspective, as Doug mentioned, we've avoided
>> >> > releasing projects
>> >> > in any kind of built form. This was also one of the concerns I raised
>> >> > when
>> >> > working on the proposal to support other programming languages. The
>> >> > problem of
>> >> > releasing built images goes beyond the infrastructure requirements.
>> >> > It's the
>> >> > message and the guarantees implied with the built product itself that
>> >> > are the
>> >> > concern here. And I tend to agree with Doug that this might be a
>> >> > problem for us
>> >> > as a community. Unfortunately, putting your name, Michal, as contact
>> >> > point is
>> >> > not enough. Kolla is not the only project producing container images
>> >> > and we need
>> >> > to be consistent in the way we release these images.
>> >> >
>> >> > Nothing prevents people for building their own images and uploading
>> >> > them to
>> >> > dockerhub. Having this as part of the OpenStack's pipeline is a
>> >> > problem.
>> >>
>> >> I totally subscribe to the concerns around publishing binaries (under
>> >> any form), and the expectations in terms of security maintenance that
>> >> it
>> >> would set on the publisher. At the same time, we need to have images
>> >> available, for convenience and testing. So what is the best way to
>> >> achieve that without setting strong security maintenance expectations
>> >> for the OpenStack community ? We have several options:
>> >>
>> >> 1/ Have third-parties publish images
>> >> It is the current situation. The issue is that the Kolla team (and
>> >> likely others) would rather automate the process and use OpenStack
>> >> infrastructure for it.
>> >>
>> >> 2/ Have third-parties publish images, but through OpenStack infra
>> >> This would allow to automate the process, but it would be a bit weird
>> >> to
>> >> use common infra resources to publish in a private repo.
>> >>
>> >> 3/ Publish transient (per-commit or daily) images
>> >> A "daily build" (especially if you replace it every day) would set
>> >> relatively-limited expectations in terms of maintenance. It would end
>> >> up
>> >> picking up security updates in upstream layers, even if not
>> >> immediately.
>> >>
>> >> 4/ Publish images and own them
>> >> Staff release / VMT / stable team in a way that lets us properly own
>> >> those images and publish them officially.
>> >>
>> >> Personally I think (4) is not realistic. I think we could make (3)
>> >> work,
>> >> and I prefer it to (2). If all else fails, we should keep (1).
>> >>
>> >
>> > At the forum we talked about putting test images on a "private"
>> > repository hosted on openstack.org somewhere. I think that's option
>> > 3 from your list?
>> >
>> > Paul may be able to shed more light on the details of the technology
>> > (maybe it's just an Apache-served repo, rather than a full blown
>> > instance of Docker's service, for example).
>>
>> Issue with that is
>>
>> 1. Apache served is harder to use because we want to follow docker API
>> and we'd have to reimplement it
>> 2. Running registry is single command
>> 3. If we host in in infra, in case someone actually uses it (there
>> will be people like that), that will eat up lot of network traffic
>> potentially
>> 4. With local caching of images (working already) in nodepools we
>> loose complexity of mirroring registries across nodepools
>>
>> So bottom line, having dockerhub/quay.io is simply easier.
>>
>> > Doug
>> >
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Edward Leafe]

On May 15, 2017, at 9:00 PM, Flavio Percoco  wrote:

> [huge snip]

Thank you! We don’t need 50K of repeated text in every response.

-- Ed Leafe





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Flavio Percoco]

On 16/05/17 09:45 -0400, Doug Hellmann wrote:

Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400:

On 15/05/17 11:49 -0700, Michał Jastrzębski wrote:
>On 15 May 2017 at 11:19, Davanum Srinivas  wrote:
>> Sorry for the top post, Michal, Can you please clarify a couple of things:
>>
>> 1) Can folks install just one or two services for their specific scenario?
>
>Yes, that's more of a kolla-ansible feature and require a little bit
>of ansible know-how, but entirely possible. Kolla-k8s is built to
>allow maximum flexibility in that space.
>
>> 2) Can the container images from kolla be run on bare docker daemon?
>
>Yes, but they need to either override our default CMD (kolla_start) or
>provide ENVs requred by it, not a huge deal
>
>> 3) Can someone take the kolla container images from say dockerhub and
>> use it without the Kolla framework?
>
>Yes, there is no such thing as kolla framework really. Our images
>follow stable ABI and they can be deployed by any deploy mechanism
>that will follow it. We have several users who wrote their own deploy
>mechanism from scratch.
>
>Containers are just blobs with binaries in it. Little things that we
>add are kolla_start script to allow our config file management and
>some custom startup scripts for things like mariadb to help with
>bootstrapping, both are entirely optional.

Just as a bonus example, TripleO is currently using kolla images. They used to
be vanilla and they are not anymore but only because TripleO depends on puppet
being in the image, which has nothing to do with kolla.

Flavio



When you say "using kolla images," what do you mean? In upstream
CI tests? On contributors' dev/test systems? Production deployments?


All of them. Note that TripleO now builds its own "kolla images" (it uses the
kolla Dockerfiles and kolla-build) because the dependency of puppet. When I
said, TripleO uses kolla images was intended to answer Dims question on whether
these images (or Dockerfiles) can be consumed by other projects.

Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Sam Yaple]

I would like to bring up a subject that hasn't really been discussed in
this thread yet, forgive me if I missed an email mentioning this.

What I personally would like to see is a publishing infrastructure to allow
pushing built images to an internal infra mirror/repo/registry for
consumption of internal infra jobs (deployment tools like kolla-ansible and
openstack-ansible). The images built from infra mirrors with security
turned off are perfect for testing internally to infra.

If you build images properly in infra, then you will have an image that is
not security checked (no gpg verification of packages) and completely
unverifiable. These are absolutely not images we want to push to
DockerHub/quay for obvious reasons. Security and verification being chief
among them. They are absolutely not images that should ever be run in
production and are only suited for testing. These are the only types of
images that can come out of infra.

Thanks,
SamYaple

On Tue, May 16, 2017 at 1:57 PM, Michał Jastrzębski 
wrote:

> On 16 May 2017 at 06:22, Doug Hellmann  wrote:
> > Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
> >> Flavio Percoco wrote:
> >> > From a release perspective, as Doug mentioned, we've avoided
> releasing projects
> >> > in any kind of built form. This was also one of the concerns I raised
> when
> >> > working on the proposal to support other programming languages. The
> problem of
> >> > releasing built images goes beyond the infrastructure requirements.
> It's the
> >> > message and the guarantees implied with the built product itself that
> are the
> >> > concern here. And I tend to agree with Doug that this might be a
> problem for us
> >> > as a community. Unfortunately, putting your name, Michal, as contact
> point is
> >> > not enough. Kolla is not the only project producing container images
> and we need
> >> > to be consistent in the way we release these images.
> >> >
> >> > Nothing prevents people for building their own images and uploading
> them to
> >> > dockerhub. Having this as part of the OpenStack's pipeline is a
> problem.
> >>
> >> I totally subscribe to the concerns around publishing binaries (under
> >> any form), and the expectations in terms of security maintenance that it
> >> would set on the publisher. At the same time, we need to have images
> >> available, for convenience and testing. So what is the best way to
> >> achieve that without setting strong security maintenance expectations
> >> for the OpenStack community ? We have several options:
> >>
> >> 1/ Have third-parties publish images
> >> It is the current situation. The issue is that the Kolla team (and
> >> likely others) would rather automate the process and use OpenStack
> >> infrastructure for it.
> >>
> >> 2/ Have third-parties publish images, but through OpenStack infra
> >> This would allow to automate the process, but it would be a bit weird to
> >> use common infra resources to publish in a private repo.
> >>
> >> 3/ Publish transient (per-commit or daily) images
> >> A "daily build" (especially if you replace it every day) would set
> >> relatively-limited expectations in terms of maintenance. It would end up
> >> picking up security updates in upstream layers, even if not immediately.
> >>
> >> 4/ Publish images and own them
> >> Staff release / VMT / stable team in a way that lets us properly own
> >> those images and publish them officially.
> >>
> >> Personally I think (4) is not realistic. I think we could make (3) work,
> >> and I prefer it to (2). If all else fails, we should keep (1).
> >>
> >
> > At the forum we talked about putting test images on a "private"
> > repository hosted on openstack.org somewhere. I think that's option
> > 3 from your list?
> >
> > Paul may be able to shed more light on the details of the technology
> > (maybe it's just an Apache-served repo, rather than a full blown
> > instance of Docker's service, for example).
>
> Issue with that is
>
> 1. Apache served is harder to use because we want to follow docker API
> and we'd have to reimplement it
> 2. Running registry is single command
> 3. If we host in in infra, in case someone actually uses it (there
> will be people like that), that will eat up lot of network traffic
> potentially
> 4. With local caching of images (working already) in nodepools we
> loose complexity of mirroring registries across nodepools
>
> So bottom line, having dockerhub/quay.io is simply easier.
>
> > Doug
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 06:22, Doug Hellmann  wrote:
> Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
>> Flavio Percoco wrote:
>> > From a release perspective, as Doug mentioned, we've avoided releasing 
>> > projects
>> > in any kind of built form. This was also one of the concerns I raised when
>> > working on the proposal to support other programming languages. The 
>> > problem of
>> > releasing built images goes beyond the infrastructure requirements. It's 
>> > the
>> > message and the guarantees implied with the built product itself that are 
>> > the
>> > concern here. And I tend to agree with Doug that this might be a problem 
>> > for us
>> > as a community. Unfortunately, putting your name, Michal, as contact point 
>> > is
>> > not enough. Kolla is not the only project producing container images and 
>> > we need
>> > to be consistent in the way we release these images.
>> >
>> > Nothing prevents people for building their own images and uploading them to
>> > dockerhub. Having this as part of the OpenStack's pipeline is a problem.
>>
>> I totally subscribe to the concerns around publishing binaries (under
>> any form), and the expectations in terms of security maintenance that it
>> would set on the publisher. At the same time, we need to have images
>> available, for convenience and testing. So what is the best way to
>> achieve that without setting strong security maintenance expectations
>> for the OpenStack community ? We have several options:
>>
>> 1/ Have third-parties publish images
>> It is the current situation. The issue is that the Kolla team (and
>> likely others) would rather automate the process and use OpenStack
>> infrastructure for it.
>>
>> 2/ Have third-parties publish images, but through OpenStack infra
>> This would allow to automate the process, but it would be a bit weird to
>> use common infra resources to publish in a private repo.
>>
>> 3/ Publish transient (per-commit or daily) images
>> A "daily build" (especially if you replace it every day) would set
>> relatively-limited expectations in terms of maintenance. It would end up
>> picking up security updates in upstream layers, even if not immediately.
>>
>> 4/ Publish images and own them
>> Staff release / VMT / stable team in a way that lets us properly own
>> those images and publish them officially.
>>
>> Personally I think (4) is not realistic. I think we could make (3) work,
>> and I prefer it to (2). If all else fails, we should keep (1).
>>
>
> At the forum we talked about putting test images on a "private"
> repository hosted on openstack.org somewhere. I think that's option
> 3 from your list?
>
> Paul may be able to shed more light on the details of the technology
> (maybe it's just an Apache-served repo, rather than a full blown
> instance of Docker's service, for example).

Issue with that is

1. Apache served is harder to use because we want to follow docker API
and we'd have to reimplement it
2. Running registry is single command
3. If we host in in infra, in case someone actually uses it (there
will be people like that), that will eat up lot of network traffic
potentially
4. With local caching of images (working already) in nodepools we
loose complexity of mirroring registries across nodepools

So bottom line, having dockerhub/quay.io is simply easier.

> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 06:22, Doug Hellmann  wrote:
> Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
>> Flavio Percoco wrote:
>> > From a release perspective, as Doug mentioned, we've avoided releasing 
>> > projects
>> > in any kind of built form. This was also one of the concerns I raised when
>> > working on the proposal to support other programming languages. The 
>> > problem of
>> > releasing built images goes beyond the infrastructure requirements. It's 
>> > the
>> > message and the guarantees implied with the built product itself that are 
>> > the
>> > concern here. And I tend to agree with Doug that this might be a problem 
>> > for us
>> > as a community. Unfortunately, putting your name, Michal, as contact point 
>> > is
>> > not enough. Kolla is not the only project producing container images and 
>> > we need
>> > to be consistent in the way we release these images.
>> >
>> > Nothing prevents people for building their own images and uploading them to
>> > dockerhub. Having this as part of the OpenStack's pipeline is a problem.
>>
>> I totally subscribe to the concerns around publishing binaries (under
>> any form), and the expectations in terms of security maintenance that it
>> would set on the publisher. At the same time, we need to have images
>> available, for convenience and testing. So what is the best way to
>> achieve that without setting strong security maintenance expectations
>> for the OpenStack community ? We have several options:
>>
>> 1/ Have third-parties publish images
>> It is the current situation. The issue is that the Kolla team (and
>> likely others) would rather automate the process and use OpenStack
>> infrastructure for it.
>>
>> 2/ Have third-parties publish images, but through OpenStack infra
>> This would allow to automate the process, but it would be a bit weird to
>> use common infra resources to publish in a private repo.
>>
>> 3/ Publish transient (per-commit or daily) images
>> A "daily build" (especially if you replace it every day) would set
>> relatively-limited expectations in terms of maintenance. It would end up
>> picking up security updates in upstream layers, even if not immediately.
>>
>> 4/ Publish images and own them
>> Staff release / VMT / stable team in a way that lets us properly own
>> those images and publish them officially.
>>
>> Personally I think (4) is not realistic. I think we could make (3) work,
>> and I prefer it to (2). If all else fails, we should keep (1).
>>
>
> At the forum we talked about putting test images on a "private"
> repository hosted on openstack.org somewhere. I think that's option
> 3 from your list?
>
> Paul may be able to shed more light on the details of the technology
> (maybe it's just an Apache-served repo, rather than a full blown
> instance of Docker's service, for example).
>
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 16 May 2017 at 06:20, Flavio Percoco  wrote:
> On 16/05/17 14:08 +0200, Thierry Carrez wrote:
>>
>> Flavio Percoco wrote:
>>>
>>> From a release perspective, as Doug mentioned, we've avoided releasing
>>> projects
>>> in any kind of built form. This was also one of the concerns I raised
>>> when
>>> working on the proposal to support other programming languages. The
>>> problem of
>>> releasing built images goes beyond the infrastructure requirements. It's
>>> the
>>> message and the guarantees implied with the built product itself that are
>>> the
>>> concern here. And I tend to agree with Doug that this might be a problem
>>> for us
>>> as a community. Unfortunately, putting your name, Michal, as contact
>>> point is
>>> not enough. Kolla is not the only project producing container images and
>>> we need
>>> to be consistent in the way we release these images.
>>>
>>> Nothing prevents people for building their own images and uploading them
>>> to
>>> dockerhub. Having this as part of the OpenStack's pipeline is a problem.
>>
>>
>> I totally subscribe to the concerns around publishing binaries (under
>> any form), and the expectations in terms of security maintenance that it
>> would set on the publisher. At the same time, we need to have images
>> available, for convenience and testing. So what is the best way to
>> achieve that without setting strong security maintenance expectations
>> for the OpenStack community ? We have several options:
>>
>> 1/ Have third-parties publish images
>> It is the current situation. The issue is that the Kolla team (and
>> likely others) would rather automate the process and use OpenStack
>> infrastructure for it.
>>
>> 2/ Have third-parties publish images, but through OpenStack infra
>> This would allow to automate the process, but it would be a bit weird to
>> use common infra resources to publish in a private repo.
>>
>> 3/ Publish transient (per-commit or daily) images
>> A "daily build" (especially if you replace it every day) would set
>> relatively-limited expectations in terms of maintenance. It would end up
>> picking up security updates in upstream layers, even if not immediately.
>>
>> 4/ Publish images and own them
>> Staff release / VMT / stable team in a way that lets us properly own
>> those images and publish them officially.
>>
>> Personally I think (4) is not realistic. I think we could make (3) work,
>> and I prefer it to (2). If all else fails, we should keep (1).
>
>
> Agreed #4 is a bit unrealistic.
>
> Not sure I understand the difference between #2 and #3. Is it just the
> cadence?
>
> I'd prefer for these builds to have a daily cadence because it sets the
> expectations w.r.t maintenance right: "These images are daily builds and not
> certified releases. For stable builds you're better off building it
> yourself"

And daily builds are exactly what I wanted in the first place:) We
probably will keep publishing release packages too, but we can be so
called 3rd party. I also agree [4] is completely unrealistic and I
would be against putting such heavy burden of responsibility on any
community, including Kolla.

While daily cadence will send message that it's not stable, truth will
be that it will be more stable than what people would normally build
locally (again, it passes more gates), but I'm totally fine in not
saying that and let people decide how they want to use it.

So, can we move on with implementation?

Thanks!
Michal

>
> Flavio
>
> --
> @flaper87
> Flavio Percoco
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2017-05-15 21:50:23 -0400:
> On 15/05/17 11:49 -0700, Michał Jastrzębski wrote:
> >On 15 May 2017 at 11:19, Davanum Srinivas  wrote:
> >> Sorry for the top post, Michal, Can you please clarify a couple of things:
> >>
> >> 1) Can folks install just one or two services for their specific scenario?
> >
> >Yes, that's more of a kolla-ansible feature and require a little bit
> >of ansible know-how, but entirely possible. Kolla-k8s is built to
> >allow maximum flexibility in that space.
> >
> >> 2) Can the container images from kolla be run on bare docker daemon?
> >
> >Yes, but they need to either override our default CMD (kolla_start) or
> >provide ENVs requred by it, not a huge deal
> >
> >> 3) Can someone take the kolla container images from say dockerhub and
> >> use it without the Kolla framework?
> >
> >Yes, there is no such thing as kolla framework really. Our images
> >follow stable ABI and they can be deployed by any deploy mechanism
> >that will follow it. We have several users who wrote their own deploy
> >mechanism from scratch.
> >
> >Containers are just blobs with binaries in it. Little things that we
> >add are kolla_start script to allow our config file management and
> >some custom startup scripts for things like mariadb to help with
> >bootstrapping, both are entirely optional.
> 
> Just as a bonus example, TripleO is currently using kolla images. They used to
> be vanilla and they are not anymore but only because TripleO depends on puppet
> being in the image, which has nothing to do with kolla.
> 
> Flavio
> 

When you say "using kolla images," what do you mean? In upstream
CI tests? On contributors' dev/test systems? Production deployments?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

This is one of those areas that was shared understanding for a long
time, and seems less "shared" now that we've grown and added new
projects to the community.  I intended to prepare a governance
resolution *after* having some public discussion, so that we can
restore that common understanding through documentation. I didn't
prepare the resolution as a first step, because if the consensus
is that we've changed our collective minds about whether publishing
binary artifacts is a good idea then the wording of the resolution
needs to reflect that.

Doug

Excerpts from Davanum Srinivas (dims)'s message of 2017-05-16 09:25:56 -0400:
> Steve,
> 
> We should not always ask "if this is a ruling from the TC", the
> default is that it's a discussion/exploration. If it is a "ruling", it
> won't be on a ML thread.
> 
> Thanks,
> Dims
> 
> On Tue, May 16, 2017 at 9:22 AM, Steven Dake (stdake) <std...@cisco.com> 
> wrote:
> > Dims,
> >
> > The [tc] was in the subject tag, and the message was represented as 
> > indicating some TC directive and has had several tc members comment on the 
> > thread.  I did nothing wrong.
> >
> > Regards
> > -steve
> >
> >
> > -Original Message-
> > From: Davanum Srinivas <dava...@gmail.com>
> > Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> > <openstack-dev@lists.openstack.org>
> > Date: Tuesday, May 16, 2017 at 4:34 AM
> > To: "OpenStack Development Mailing List (not for usage questions)" 
> > <openstack-dev@lists.openstack.org>
> > Subject: Re: [openstack-dev] 
> > [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
> >  do we want to be publishing binary container images?
> >
> > Why drag TC into this discussion Steven? If the TC has something to
> > say, it will be in the form of a resolution with topic "formal-vote".
> > So please Stop!
> >
> > Thanks,
> > Dims
> >
> > On Tue, May 16, 2017 at 12:22 AM, Steven Dake (stdake) 
> > <std...@cisco.com> wrote:
> > > Flavio,
> > >
> > > Forgive the top post – outlook ftw.
> > >
> > > I understand the concerns raised in this thread.  It is unclear if 
> > this thread is the feeling of two TC members or enough TC members care 
> > deeply about this issue to permanently limit OpenStack big tent projects’ 
> > ability to generate container images in various external artifact storage 
> > systems.  The point of discussion I see effectively raised in this thread 
> > is “OpenStack infra will not push images to dockerhub”.
> > >
> > > I’d like clarification if this is a ruling from the TC, or simply an 
> > exploratory discussion.
> > >
> > > If it is exploratory, it is prudent that OpenStack projects not be 
> > blocked by debate on this issue until the TC has made such ruling as to 
> > banning the creation of container images via OpenStack infrastructure.
> > >
> > > Regards
> >     > -steve
> >     >
> > > -----Original Message-----
> >     > From: Flavio Percoco <fla...@redhat.com>
> > > Reply-To: "OpenStack Development Mailing List (not for usage 
> > questions)" <openstack-dev@lists.openstack.org>
> > > Date: Monday, May 15, 2017 at 7:00 PM
> > > To: "OpenStack Development Mailing List (not for usage questions)" 
> > <openstack-dev@lists.openstack.org>
> > > Subject: Re: [openstack-dev] 
> > [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
> >  do we want to be publishing binary container images?
> > >
> > > On 15/05/17 12:32 -0700, Michał Jastrzębski wrote:
> > > >On 15 May 2017 at 12:12, Doug Hellmann <d...@doughellmann.com> 
> > wrote:
> > >
> > > [huge snip]
> > >
> > > >>> > I'm raising the issue here to get some more input into how 
> > to
> > > >>> > proceed. Do other people think this concern is overblown? 
> > Can we
> > > >>> > mitigate the risk by communicating through metadata for the 
> > images?
> > > >>> > Should we stick to publishing build instructions 
> > (Dockerfiles, or
> > > >>> > whatever) instead of binary images? Are there other options 
> > I haven't
> > > &

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Davanum Srinivas]

Steve,

We should not always ask "if this is a ruling from the TC", the
default is that it's a discussion/exploration. If it is a "ruling", it
won't be on a ML thread.

Thanks,
Dims

On Tue, May 16, 2017 at 9:22 AM, Steven Dake (stdake) <std...@cisco.com> wrote:
> Dims,
>
> The [tc] was in the subject tag, and the message was represented as 
> indicating some TC directive and has had several tc members comment on the 
> thread.  I did nothing wrong.
>
> Regards
> -steve
>
>
> -Original Message-
> From: Davanum Srinivas <dava...@gmail.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Date: Tuesday, May 16, 2017 at 4:34 AM
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] 
> [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
>  do we want to be publishing binary container images?
>
> Why drag TC into this discussion Steven? If the TC has something to
> say, it will be in the form of a resolution with topic "formal-vote".
> So please Stop!
>
> Thanks,
> Dims
>
> On Tue, May 16, 2017 at 12:22 AM, Steven Dake (stdake) <std...@cisco.com> 
> wrote:
> > Flavio,
> >
> > Forgive the top post – outlook ftw.
> >
> > I understand the concerns raised in this thread.  It is unclear if this 
> thread is the feeling of two TC members or enough TC members care deeply 
> about this issue to permanently limit OpenStack big tent projects’ ability to 
> generate container images in various external artifact storage systems.  The 
> point of discussion I see effectively raised in this thread is “OpenStack 
> infra will not push images to dockerhub”.
> >
> > I’d like clarification if this is a ruling from the TC, or simply an 
> exploratory discussion.
> >
> > If it is exploratory, it is prudent that OpenStack projects not be 
> blocked by debate on this issue until the TC has made such ruling as to 
> banning the creation of container images via OpenStack infrastructure.
> >
> > Regards
> > -steve
> >
> > -Original Message-
> > From: Flavio Percoco <fla...@redhat.com>
> > Reply-To: "OpenStack Development Mailing List (not for usage 
> questions)" <openstack-dev@lists.openstack.org>
> > Date: Monday, May 15, 2017 at 7:00 PM
> > To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> > Subject: Re: [openstack-dev] 
> [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
>  do we want to be publishing binary container images?
> >
> > On 15/05/17 12:32 -0700, Michał Jastrzębski wrote:
> > >On 15 May 2017 at 12:12, Doug Hellmann <d...@doughellmann.com> 
> wrote:
> >
> > [huge snip]
> >
> > >>> > I'm raising the issue here to get some more input into how to
> > >>> > proceed. Do other people think this concern is overblown? Can 
> we
> > >>> > mitigate the risk by communicating through metadata for the 
> images?
> > >>> > Should we stick to publishing build instructions 
> (Dockerfiles, or
> > >>> > whatever) instead of binary images? Are there other options I 
> haven't
> > >>> > mentioned?
> > >>>
> > >>> Today we do publish build instructions, that's what Kolla is. 
> We also
> > >>> publish built containers already, just we do it manually on 
> release
> > >>> today. If we decide to block it, I assume we should stop doing 
> that
> > >>> too? That will hurt users who uses this piece of Kolla, and I'd 
> hate
> > >>> to hurt our users:(
> > >>
> > >> Well, that's the question. Today we have teams publishing those
> > >> images themselves, right? And the proposal is to have infra do 
> it?
> > >> That change could be construed to imply that there is more of a
> > >> relationship with the images and the rest of the community 
> (remember,
> > >> folks outside of the main community activities do not always make
> > >> the same distinctions we do about teams). 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Sean Dague]

On 05/16/2017 09:24 AM, Doug Hellmann wrote:
> Excerpts from Luigi Toscano's message of 2017-05-16 11:50:53 +0200:
>> On Monday, 15 May 2017 21:12:16 CEST Doug Hellmann wrote:
>>> Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700:
>>>
 On 15 May 2017 at 10:34, Doug Hellmann  wrote:
> I'm raising the issue here to get some more input into how to
> proceed. Do other people think this concern is overblown? Can we
> mitigate the risk by communicating through metadata for the images?
> Should we stick to publishing build instructions (Dockerfiles, or
> whatever) instead of binary images? Are there other options I haven't
> mentioned?

 Today we do publish build instructions, that's what Kolla is. We also
 publish built containers already, just we do it manually on release
 today. If we decide to block it, I assume we should stop doing that
 too? That will hurt users who uses this piece of Kolla, and I'd hate
 to hurt our users:(
>>>
>>> Well, that's the question. Today we have teams publishing those
>>> images themselves, right? And the proposal is to have infra do it?
>>> That change could be construed to imply that there is more of a
>>> relationship with the images and the rest of the community (remember,
>>> folks outside of the main community activities do not always make
>>> the same distinctions we do about teams). So, before we go ahead
>>> with that, I want to make sure that we all have a chance to discuss
>>> the policy change and its implications.
>>
>> Sorry for hijacking the thread, but we have a similar scenario for example 
>> in 
>> Sahara. It is about full VM images containing Hadoop/Spark/other_big_data 
>> stuff, and not containers, but it's looks really the same.
>> So far ready-made images have been published under 
>> http://sahara-files.mirantis.com/images/upstream/, but we are looking to 
>> have them hosted on 
>> openstack.org, just like other artifacts. 
>>
>> We asked about this few days ago on openstack-infra@, but no answer so far 
>> (the Summit didn't help):
>>
>> http://lists.openstack.org/pipermail/openstack-infra/2017-April/005312.html
>>
>> I think that the answer to the question raised in this thread is definitely 
>> going to be relevant for our use case.
>>
>> Ciao
> 
> Thanks for raising this. I think the same concerns apply to VM images.

Agreed.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Flavio Percoco]

On 16/05/17 14:08 +0200, Thierry Carrez wrote:

Flavio Percoco wrote:

From a release perspective, as Doug mentioned, we've avoided releasing projects
in any kind of built form. This was also one of the concerns I raised when
working on the proposal to support other programming languages. The problem of
releasing built images goes beyond the infrastructure requirements. It's the
message and the guarantees implied with the built product itself that are the
concern here. And I tend to agree with Doug that this might be a problem for us
as a community. Unfortunately, putting your name, Michal, as contact point is
not enough. Kolla is not the only project producing container images and we need
to be consistent in the way we release these images.

Nothing prevents people for building their own images and uploading them to
dockerhub. Having this as part of the OpenStack's pipeline is a problem.


I totally subscribe to the concerns around publishing binaries (under
any form), and the expectations in terms of security maintenance that it
would set on the publisher. At the same time, we need to have images
available, for convenience and testing. So what is the best way to
achieve that without setting strong security maintenance expectations
for the OpenStack community ? We have several options:

1/ Have third-parties publish images
It is the current situation. The issue is that the Kolla team (and
likely others) would rather automate the process and use OpenStack
infrastructure for it.

2/ Have third-parties publish images, but through OpenStack infra
This would allow to automate the process, but it would be a bit weird to
use common infra resources to publish in a private repo.

3/ Publish transient (per-commit or daily) images
A "daily build" (especially if you replace it every day) would set
relatively-limited expectations in terms of maintenance. It would end up
picking up security updates in upstream layers, even if not immediately.

4/ Publish images and own them
Staff release / VMT / stable team in a way that lets us properly own
those images and publish them officially.

Personally I think (4) is not realistic. I think we could make (3) work,
and I prefer it to (2). If all else fails, we should keep (1).


Agreed #4 is a bit unrealistic.

Not sure I understand the difference between #2 and #3. Is it just the cadence?

I'd prefer for these builds to have a daily cadence because it sets the
expectations w.r.t maintenance right: "These images are daily builds and not
certified releases. For stable builds you're better off building it yourself"

Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Luigi Toscano's message of 2017-05-16 11:50:53 +0200:
> On Monday, 15 May 2017 21:12:16 CEST Doug Hellmann wrote:
> > Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700:
> > 
> > > On 15 May 2017 at 10:34, Doug Hellmann  wrote:
> > > > I'm raising the issue here to get some more input into how to
> > > > proceed. Do other people think this concern is overblown? Can we
> > > > mitigate the risk by communicating through metadata for the images?
> > > > Should we stick to publishing build instructions (Dockerfiles, or
> > > > whatever) instead of binary images? Are there other options I haven't
> > > > mentioned?
> > > 
> > > Today we do publish build instructions, that's what Kolla is. We also
> > > publish built containers already, just we do it manually on release
> > > today. If we decide to block it, I assume we should stop doing that
> > > too? That will hurt users who uses this piece of Kolla, and I'd hate
> > > to hurt our users:(
> > 
> > Well, that's the question. Today we have teams publishing those
> > images themselves, right? And the proposal is to have infra do it?
> > That change could be construed to imply that there is more of a
> > relationship with the images and the rest of the community (remember,
> > folks outside of the main community activities do not always make
> > the same distinctions we do about teams). So, before we go ahead
> > with that, I want to make sure that we all have a chance to discuss
> > the policy change and its implications.
> 
> Sorry for hijacking the thread, but we have a similar scenario for example in 
> Sahara. It is about full VM images containing Hadoop/Spark/other_big_data 
> stuff, and not containers, but it's looks really the same.
> So far ready-made images have been published under 
> http://sahara-files.mirantis.com/images/upstream/, but we are looking to have 
> them hosted on 
> openstack.org, just like other artifacts. 
> 
> We asked about this few days ago on openstack-infra@, but no answer so far 
> (the Summit didn't help):
> 
> http://lists.openstack.org/pipermail/openstack-infra/2017-April/005312.html
> 
> I think that the answer to the question raised in this thread is definitely 
> going to be relevant for our use case.
> 
> Ciao

Thanks for raising this. I think the same concerns apply to VM images.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2017-05-16 14:08:07 +0200:
> Flavio Percoco wrote:
> > From a release perspective, as Doug mentioned, we've avoided releasing 
> > projects
> > in any kind of built form. This was also one of the concerns I raised when
> > working on the proposal to support other programming languages. The problem 
> > of
> > releasing built images goes beyond the infrastructure requirements. It's the
> > message and the guarantees implied with the built product itself that are 
> > the
> > concern here. And I tend to agree with Doug that this might be a problem 
> > for us
> > as a community. Unfortunately, putting your name, Michal, as contact point 
> > is
> > not enough. Kolla is not the only project producing container images and we 
> > need
> > to be consistent in the way we release these images.
> > 
> > Nothing prevents people for building their own images and uploading them to
> > dockerhub. Having this as part of the OpenStack's pipeline is a problem.
> 
> I totally subscribe to the concerns around publishing binaries (under
> any form), and the expectations in terms of security maintenance that it
> would set on the publisher. At the same time, we need to have images
> available, for convenience and testing. So what is the best way to
> achieve that without setting strong security maintenance expectations
> for the OpenStack community ? We have several options:
> 
> 1/ Have third-parties publish images
> It is the current situation. The issue is that the Kolla team (and
> likely others) would rather automate the process and use OpenStack
> infrastructure for it.
> 
> 2/ Have third-parties publish images, but through OpenStack infra
> This would allow to automate the process, but it would be a bit weird to
> use common infra resources to publish in a private repo.
> 
> 3/ Publish transient (per-commit or daily) images
> A "daily build" (especially if you replace it every day) would set
> relatively-limited expectations in terms of maintenance. It would end up
> picking up security updates in upstream layers, even if not immediately.
> 
> 4/ Publish images and own them
> Staff release / VMT / stable team in a way that lets us properly own
> those images and publish them officially.
> 
> Personally I think (4) is not realistic. I think we could make (3) work,
> and I prefer it to (2). If all else fails, we should keep (1).
> 

At the forum we talked about putting test images on a "private"
repository hosted on openstack.org somewhere. I think that's option
3 from your list?

Paul may be able to shed more light on the details of the technology
(maybe it's just an Apache-served repo, rather than a full blown
instance of Docker's service, for example).

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Steven Dake (stdake)]

Dims,

The [tc] was in the subject tag, and the message was represented as indicating 
some TC directive and has had several tc members comment on the thread.  I did 
nothing wrong.

Regards
-steve


-Original Message-
From: Davanum Srinivas <dava...@gmail.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Date: Tuesday, May 16, 2017 at 4:34 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

Why drag TC into this discussion Steven? If the TC has something to
say, it will be in the form of a resolution with topic "formal-vote".
So please Stop!

Thanks,
Dims

On Tue, May 16, 2017 at 12:22 AM, Steven Dake (stdake) <std...@cisco.com> 
wrote:
> Flavio,
>
> Forgive the top post – outlook ftw.
>
> I understand the concerns raised in this thread.  It is unclear if this 
thread is the feeling of two TC members or enough TC members care deeply about 
this issue to permanently limit OpenStack big tent projects’ ability to 
generate container images in various external artifact storage systems.  The 
point of discussion I see effectively raised in this thread is “OpenStack infra 
will not push images to dockerhub”.
>
> I’d like clarification if this is a ruling from the TC, or simply an 
exploratory discussion.
>
> If it is exploratory, it is prudent that OpenStack projects not be 
blocked by debate on this issue until the TC has made such ruling as to banning 
the creation of container images via OpenStack infrastructure.
>
> Regards
> -steve
>
> -Original Message-
> From: Flavio Percoco <fla...@redhat.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
> Date: Monday, May 15, 2017 at 7:00 PM
    > To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?
>
> On 15/05/17 12:32 -0700, Michał Jastrzębski wrote:
> >On 15 May 2017 at 12:12, Doug Hellmann <d...@doughellmann.com> wrote:
>
> [huge snip]
>
> >>> > I'm raising the issue here to get some more input into how to
> >>> > proceed. Do other people think this concern is overblown? Can we
> >>> > mitigate the risk by communicating through metadata for the 
images?
> >>> > Should we stick to publishing build instructions (Dockerfiles, 
or
> >>> > whatever) instead of binary images? Are there other options I 
haven't
> >>> > mentioned?
> >>>
> >>> Today we do publish build instructions, that's what Kolla is. We 
also
> >>> publish built containers already, just we do it manually on 
release
> >>> today. If we decide to block it, I assume we should stop doing 
that
> >>> too? That will hurt users who uses this piece of Kolla, and I'd 
hate
> >>> to hurt our users:(
> >>
> >> Well, that's the question. Today we have teams publishing those
> >> images themselves, right? And the proposal is to have infra do it?
> >> That change could be construed to imply that there is more of a
> >> relationship with the images and the rest of the community 
(remember,
> >> folks outside of the main community activities do not always make
> >> the same distinctions we do about teams). So, before we go ahead
> >> with that, I want to make sure that we all have a chance to discuss
> >> the policy change and its implications.
> >
> >Infra as vm running with infra, but team to publish it can be Kolla
> >team. I assume we'll be responsible to keep these images healthy...
>
> I think this is the gist of the concern and I'd like us to focus on 
it.
>
> As someone that used to consume these images from kolla's dockerhub 
account
> directly, I can confirm they are useful. However, I do share Doug's 
concern and
> the impact this may have on the community.
>
> From a release perspective

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Flavio Percoco]

On 16/05/17 04:22 +, Steven Dake (stdake) wrote:

Flavio,

Forgive the top post – outlook ftw.

I understand the concerns raised in this thread.  It is unclear if this thread 
is the feeling of two TC members or enough TC members care deeply about this 
issue to permanently limit OpenStack big tent projects’ ability to generate 
container images in various external artifact storage systems.  The point of 
discussion I see effectively raised in this thread is “OpenStack infra will not 
push images to dockerhub”.

I’d like clarification if this is a ruling from the TC, or simply an 
exploratory discussion.

If it is exploratory, it is prudent that OpenStack projects not be blocked by 
debate on this issue until the TC has made such ruling as to banning the 
creation of container images via OpenStack infrastructure.


Hey Steven,

It's nothing to do with the TC. It's a release management concern and I just
happen to have an opinion on it. :)

As Doug mentioned, OpenStack has (almost) never released binaries in any form.
This doesn't mean we can't revisit this "rule" but until that happens, the
concern stands.

Flavio


Regards
-steve

-Original Message-
From: Flavio Percoco <fla...@redhat.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Date: Monday, May 15, 2017 at 7:00 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

   On 15/05/17 12:32 -0700, Michał Jastrzębski wrote:
   >On 15 May 2017 at 12:12, Doug Hellmann <d...@doughellmann.com> wrote:

   [huge snip]

   >>> > I'm raising the issue here to get some more input into how to
   >>> > proceed. Do other people think this concern is overblown? Can we
   >>> > mitigate the risk by communicating through metadata for the images?
   >>> > Should we stick to publishing build instructions (Dockerfiles, or
   >>> > whatever) instead of binary images? Are there other options I haven't
   >>> > mentioned?
   >>>
   >>> Today we do publish build instructions, that's what Kolla is. We also
   >>> publish built containers already, just we do it manually on release
   >>> today. If we decide to block it, I assume we should stop doing that
   >>> too? That will hurt users who uses this piece of Kolla, and I'd hate
   >>> to hurt our users:(
   >>
   >> Well, that's the question. Today we have teams publishing those
   >> images themselves, right? And the proposal is to have infra do it?
   >> That change could be construed to imply that there is more of a
   >> relationship with the images and the rest of the community (remember,
   >> folks outside of the main community activities do not always make
   >> the same distinctions we do about teams). So, before we go ahead
   >> with that, I want to make sure that we all have a chance to discuss
   >> the policy change and its implications.
   >
   >Infra as vm running with infra, but team to publish it can be Kolla
   >team. I assume we'll be responsible to keep these images healthy...

   I think this is the gist of the concern and I'd like us to focus on it.

   As someone that used to consume these images from kolla's dockerhub account
   directly, I can confirm they are useful. However, I do share Doug's concern 
and
   the impact this may have on the community.

   From a release perspective, as Doug mentioned, we've avoided releasing 
projects
   in any kind of built form. This was also one of the concerns I raised when
   working on the proposal to support other programming languages. The problem 
of
   releasing built images goes beyond the infrastructure requirements. It's the
   message and the guarantees implied with the built product itself that are the
   concern here. And I tend to agree with Doug that this might be a problem for 
us
   as a community. Unfortunately, putting your name, Michal, as contact point is
   not enough. Kolla is not the only project producing container images and we 
need
   to be consistent in the way we release these images.

   Nothing prevents people for building their own images and uploading them to
   dockerhub. Having this as part of the OpenStack's pipeline is a problem.

   Flavio

   P.S: note this goes against my container(ish) interests but it's a
   community-wide problem.

   --
   @flaper87
   Flavio Percoco


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
ht

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Thierry Carrez]

Flavio Percoco wrote:
> From a release perspective, as Doug mentioned, we've avoided releasing 
> projects
> in any kind of built form. This was also one of the concerns I raised when
> working on the proposal to support other programming languages. The problem of
> releasing built images goes beyond the infrastructure requirements. It's the
> message and the guarantees implied with the built product itself that are the
> concern here. And I tend to agree with Doug that this might be a problem for 
> us
> as a community. Unfortunately, putting your name, Michal, as contact point is
> not enough. Kolla is not the only project producing container images and we 
> need
> to be consistent in the way we release these images.
> 
> Nothing prevents people for building their own images and uploading them to
> dockerhub. Having this as part of the OpenStack's pipeline is a problem.

I totally subscribe to the concerns around publishing binaries (under
any form), and the expectations in terms of security maintenance that it
would set on the publisher. At the same time, we need to have images
available, for convenience and testing. So what is the best way to
achieve that without setting strong security maintenance expectations
for the OpenStack community ? We have several options:

1/ Have third-parties publish images
It is the current situation. The issue is that the Kolla team (and
likely others) would rather automate the process and use OpenStack
infrastructure for it.

2/ Have third-parties publish images, but through OpenStack infra
This would allow to automate the process, but it would be a bit weird to
use common infra resources to publish in a private repo.

3/ Publish transient (per-commit or daily) images
A "daily build" (especially if you replace it every day) would set
relatively-limited expectations in terms of maintenance. It would end up
picking up security updates in upstream layers, even if not immediately.

4/ Publish images and own them
Staff release / VMT / stable team in a way that lets us properly own
those images and publish them officially.

Personally I think (4) is not realistic. I think we could make (3) work,
and I prefer it to (2). If all else fails, we should keep (1).

-- 
Thierry Carrez (ttx)



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Davanum Srinivas]

Why drag TC into this discussion Steven? If the TC has something to
say, it will be in the form of a resolution with topic "formal-vote".
So please Stop!

Thanks,
Dims

On Tue, May 16, 2017 at 12:22 AM, Steven Dake (stdake) <std...@cisco.com> wrote:
> Flavio,
>
> Forgive the top post – outlook ftw.
>
> I understand the concerns raised in this thread.  It is unclear if this 
> thread is the feeling of two TC members or enough TC members care deeply 
> about this issue to permanently limit OpenStack big tent projects’ ability to 
> generate container images in various external artifact storage systems.  The 
> point of discussion I see effectively raised in this thread is “OpenStack 
> infra will not push images to dockerhub”.
>
> I’d like clarification if this is a ruling from the TC, or simply an 
> exploratory discussion.
>
> If it is exploratory, it is prudent that OpenStack projects not be blocked by 
> debate on this issue until the TC has made such ruling as to banning the 
> creation of container images via OpenStack infrastructure.
>
> Regards
> -steve
>
> -Original Message-
> From: Flavio Percoco <fla...@redhat.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Date: Monday, May 15, 2017 at 7:00 PM
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] 
> [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
>  do we want to be publishing binary container images?
>
> On 15/05/17 12:32 -0700, Michał Jastrzębski wrote:
> >On 15 May 2017 at 12:12, Doug Hellmann <d...@doughellmann.com> wrote:
>
> [huge snip]
>
> >>> > I'm raising the issue here to get some more input into how to
> >>> > proceed. Do other people think this concern is overblown? Can we
> >>> > mitigate the risk by communicating through metadata for the images?
> >>> > Should we stick to publishing build instructions (Dockerfiles, or
> >>> > whatever) instead of binary images? Are there other options I 
> haven't
> >>> > mentioned?
> >>>
> >>> Today we do publish build instructions, that's what Kolla is. We also
> >>> publish built containers already, just we do it manually on release
> >>> today. If we decide to block it, I assume we should stop doing that
> >>> too? That will hurt users who uses this piece of Kolla, and I'd hate
> >>> to hurt our users:(
> >>
> >> Well, that's the question. Today we have teams publishing those
> >> images themselves, right? And the proposal is to have infra do it?
> >> That change could be construed to imply that there is more of a
> >> relationship with the images and the rest of the community (remember,
> >> folks outside of the main community activities do not always make
> >> the same distinctions we do about teams). So, before we go ahead
> >> with that, I want to make sure that we all have a chance to discuss
> >> the policy change and its implications.
> >
> >Infra as vm running with infra, but team to publish it can be Kolla
> >team. I assume we'll be responsible to keep these images healthy...
>
> I think this is the gist of the concern and I'd like us to focus on it.
>
> As someone that used to consume these images from kolla's dockerhub 
> account
> directly, I can confirm they are useful. However, I do share Doug's 
> concern and
> the impact this may have on the community.
>
> From a release perspective, as Doug mentioned, we've avoided releasing 
> projects
> in any kind of built form. This was also one of the concerns I raised when
> working on the proposal to support other programming languages. The 
> problem of
> releasing built images goes beyond the infrastructure requirements. It's 
> the
> message and the guarantees implied with the built product itself that are 
> the
> concern here. And I tend to agree with Doug that this might be a problem 
> for us
> as a community. Unfortunately, putting your name, Michal, as contact 
> point is
> not enough. Kolla is not the only project producing container images and 
> we need
> to be consistent in the way we release these images.
>
> Nothing prevents people for building their own images and uploading them 
> to
> dockerhub. Having this as part of the OpenStack's pipeli

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Luigi Toscano]

On Monday, 15 May 2017 21:12:16 CEST Doug Hellmann wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700:
> 
> > On 15 May 2017 at 10:34, Doug Hellmann  wrote:
> > > I'm raising the issue here to get some more input into how to
> > > proceed. Do other people think this concern is overblown? Can we
> > > mitigate the risk by communicating through metadata for the images?
> > > Should we stick to publishing build instructions (Dockerfiles, or
> > > whatever) instead of binary images? Are there other options I haven't
> > > mentioned?
> > 
> > Today we do publish build instructions, that's what Kolla is. We also
> > publish built containers already, just we do it manually on release
> > today. If we decide to block it, I assume we should stop doing that
> > too? That will hurt users who uses this piece of Kolla, and I'd hate
> > to hurt our users:(
> 
> Well, that's the question. Today we have teams publishing those
> images themselves, right? And the proposal is to have infra do it?
> That change could be construed to imply that there is more of a
> relationship with the images and the rest of the community (remember,
> folks outside of the main community activities do not always make
> the same distinctions we do about teams). So, before we go ahead
> with that, I want to make sure that we all have a chance to discuss
> the policy change and its implications.

Sorry for hijacking the thread, but we have a similar scenario for example in 
Sahara. It is about full VM images containing Hadoop/Spark/other_big_data 
stuff, and not containers, but it's looks really the same.
So far ready-made images have been published under 
http://sahara-files.mirantis.com/images/upstream/, but we are looking to have 
them hosted on 
openstack.org, just like other artifacts. 

We asked about this few days ago on openstack-infra@, but no answer so far 
(the Summit didn't help):

http://lists.openstack.org/pipermail/openstack-infra/2017-April/005312.html

I think that the answer to the question raised in this thread is definitely 
going to be relevant for our use case.

Ciao
-- 
Luigi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Steven Dake (stdake)]

Flavio,

Forgive the top post – outlook ftw.

I understand the concerns raised in this thread.  It is unclear if this thread 
is the feeling of two TC members or enough TC members care deeply about this 
issue to permanently limit OpenStack big tent projects’ ability to generate 
container images in various external artifact storage systems.  The point of 
discussion I see effectively raised in this thread is “OpenStack infra will not 
push images to dockerhub”.

I’d like clarification if this is a ruling from the TC, or simply an 
exploratory discussion.

If it is exploratory, it is prudent that OpenStack projects not be blocked by 
debate on this issue until the TC has made such ruling as to banning the 
creation of container images via OpenStack infrastructure.

Regards
-steve

-Original Message-
From: Flavio Percoco <fla...@redhat.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Date: Monday, May 15, 2017 at 7:00 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] 
[tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes]
 do we want to be publishing binary container images?

On 15/05/17 12:32 -0700, Michał Jastrzębski wrote:
>On 15 May 2017 at 12:12, Doug Hellmann <d...@doughellmann.com> wrote:

[huge snip]

>>> > I'm raising the issue here to get some more input into how to
>>> > proceed. Do other people think this concern is overblown? Can we
>>> > mitigate the risk by communicating through metadata for the images?
>>> > Should we stick to publishing build instructions (Dockerfiles, or
>>> > whatever) instead of binary images? Are there other options I haven't
>>> > mentioned?
>>>
>>> Today we do publish build instructions, that's what Kolla is. We also
>>> publish built containers already, just we do it manually on release
>>> today. If we decide to block it, I assume we should stop doing that
>>> too? That will hurt users who uses this piece of Kolla, and I'd hate
>>> to hurt our users:(
>>
>> Well, that's the question. Today we have teams publishing those
>> images themselves, right? And the proposal is to have infra do it?
>> That change could be construed to imply that there is more of a
>> relationship with the images and the rest of the community (remember,
>> folks outside of the main community activities do not always make
>> the same distinctions we do about teams). So, before we go ahead
>> with that, I want to make sure that we all have a chance to discuss
>> the policy change and its implications.
>
>Infra as vm running with infra, but team to publish it can be Kolla
>team. I assume we'll be responsible to keep these images healthy...

I think this is the gist of the concern and I'd like us to focus on it.

As someone that used to consume these images from kolla's dockerhub account
directly, I can confirm they are useful. However, I do share Doug's concern 
and
the impact this may have on the community.

From a release perspective, as Doug mentioned, we've avoided releasing 
projects
in any kind of built form. This was also one of the concerns I raised when
working on the proposal to support other programming languages. The problem 
of
releasing built images goes beyond the infrastructure requirements. It's the
message and the guarantees implied with the built product itself that are 
the
concern here. And I tend to agree with Doug that this might be a problem 
for us
as a community. Unfortunately, putting your name, Michal, as contact point 
is
not enough. Kolla is not the only project producing container images and we 
need
to be consistent in the way we release these images.

Nothing prevents people for building their own images and uploading them to
dockerhub. Having this as part of the OpenStack's pipeline is a problem.

Flavio

P.S: note this goes against my container(ish) interests but it's a
community-wide problem.

-- 
@flaper87
Flavio Percoco


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Flavio Percoco]

On 15/05/17 12:32 -0700, Michał Jastrzębski wrote:

On 15 May 2017 at 12:12, Doug Hellmann  wrote:


[huge snip]


> I'm raising the issue here to get some more input into how to
> proceed. Do other people think this concern is overblown? Can we
> mitigate the risk by communicating through metadata for the images?
> Should we stick to publishing build instructions (Dockerfiles, or
> whatever) instead of binary images? Are there other options I haven't
> mentioned?

Today we do publish build instructions, that's what Kolla is. We also
publish built containers already, just we do it manually on release
today. If we decide to block it, I assume we should stop doing that
too? That will hurt users who uses this piece of Kolla, and I'd hate
to hurt our users:(


Well, that's the question. Today we have teams publishing those
images themselves, right? And the proposal is to have infra do it?
That change could be construed to imply that there is more of a
relationship with the images and the rest of the community (remember,
folks outside of the main community activities do not always make
the same distinctions we do about teams). So, before we go ahead
with that, I want to make sure that we all have a chance to discuss
the policy change and its implications.


Infra as vm running with infra, but team to publish it can be Kolla
team. I assume we'll be responsible to keep these images healthy...


I think this is the gist of the concern and I'd like us to focus on it.

As someone that used to consume these images from kolla's dockerhub account
directly, I can confirm they are useful. However, I do share Doug's concern and
the impact this may have on the community.

From a release perspective, as Doug mentioned, we've avoided releasing projects
in any kind of built form. This was also one of the concerns I raised when
working on the proposal to support other programming languages. The problem of
releasing built images goes beyond the infrastructure requirements. It's the
message and the guarantees implied with the built product itself that are the
concern here. And I tend to agree with Doug that this might be a problem for us
as a community. Unfortunately, putting your name, Michal, as contact point is
not enough. Kolla is not the only project producing container images and we need
to be consistent in the way we release these images.

Nothing prevents people for building their own images and uploading them to
dockerhub. Having this as part of the OpenStack's pipeline is a problem.

Flavio

P.S: note this goes against my container(ish) interests but it's a
community-wide problem.

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Flavio Percoco]

On 15/05/17 11:49 -0700, Michał Jastrzębski wrote:

On 15 May 2017 at 11:19, Davanum Srinivas  wrote:

Sorry for the top post, Michal, Can you please clarify a couple of things:

1) Can folks install just one or two services for their specific scenario?


Yes, that's more of a kolla-ansible feature and require a little bit
of ansible know-how, but entirely possible. Kolla-k8s is built to
allow maximum flexibility in that space.


2) Can the container images from kolla be run on bare docker daemon?


Yes, but they need to either override our default CMD (kolla_start) or
provide ENVs requred by it, not a huge deal


3) Can someone take the kolla container images from say dockerhub and
use it without the Kolla framework?


Yes, there is no such thing as kolla framework really. Our images
follow stable ABI and they can be deployed by any deploy mechanism
that will follow it. We have several users who wrote their own deploy
mechanism from scratch.

Containers are just blobs with binaries in it. Little things that we
add are kolla_start script to allow our config file management and
some custom startup scripts for things like mariadb to help with
bootstrapping, both are entirely optional.


Just as a bonus example, TripleO is currently using kolla images. They used to
be vanilla and they are not anymore but only because TripleO depends on puppet
being in the image, which has nothing to do with kolla.

Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 15 May 2017 at 12:12, Doug Hellmann  wrote:
> Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700:
>> For starters, I want to emphasize that fresh set of dockerhub images
>> was one of most requested features from Kolla on this summit and few
>> other features more or less requires readily-available docker
>> registry. Features like full release upgrade gates.
>>
>> This will have numerous benefits for users that doesn't have resources
>> to put sophisticated CI/staging env, which, I'm willing to bet, is
>> still quite significant user base. If we do it correctly (and we will
>> do it correctly), images we'll going to push will go through series of
>> gates which we have in Kolla (and will have more). So when you pull
>> image, you know that it was successfully deployed within scenerios
>> available in our gates, maybe even upgrade and increase scenerio
>> coverage later? That is a huge benefit for actual users.
>
> I have no doubt that consumers of the images would like us to keep
> creating them. We had lots of discussions last week about resource
> constraints and sustainable practices, though, and this strikes me
> as an area where we're deviating from our history in a way that
> will require more maintenance work upstream.
>
>> On 15 May 2017 at 10:34, Doug Hellmann  wrote:
>> > Last week at the Forum we had a couple of discussions about
>> > collaboration between the various teams building or consuming
>> > container images. One topic that came up was deciding how to publish
>> > images from the various teams to docker hub or other container
>> > registries. While the technical bits seem easy enough to work out,
>> > there is still the question of precedence and whether it's a good
>> > idea to do so at all.
>> >
>> > In the past, we have refrained from publishing binary packages in
>> > other formats such as debs and RPMs. (We did publish debs way back
>> > in the beginning, for testing IIRC, but switched away from them to
>> > sdists to be more inclusive.) Since then, we have said it is the
>> > responsibility of downstream consumers to build production packages,
>> > either as distributors or as a deployer that is rolling their own.
>> > We do package sdists for python libraries, push some JavaScript to
>> > the NPM registries, and have tarballs of those and a bunch of other
>> > artifacts that we build out of our release tools.  But none of those
>> > is declared as "production ready," and so the community is not
>> > sending the signal that we are responsible for maintaining them in
>> > the context of production deployments, beyond continuing to produce
>> > new releases when there are bugs.
>>
>> So for us that would mean something really hacky and bad. We are
>> community driven not company driven project. We don't have Red Hat or
>> Canonical teams behind us (we have contributors, but that's
>> different).
>
> Although I work at Red Hat, I want to make sure it's clear that my
> objection is purely related to community concerns. For this
> conversation, I'm wearing my upstream TC and Release team hats.
>
>> > Container images introduce some extra complexity, over the basic
>> > operating system style packages mentioned above. Due to the way
>> > they are constructed, they are likely to include content we don't
>> > produce ourselves (either in the form of base layers or via including
>> > build tools or other things needed when assembling the full image).
>> > That extra content means there would need to be more tracking of
>> > upstream issues (bugs, CVEs, etc.) to ensure the images are updated
>> > as needed.
>>
>> We can do this by building daily, which was the plan in fact. If we
>> build every day you have at most 24hrs old packages, CVEs and things
>> like that on non-openstack packages are still maintained by distro
>> maintainers.
>
> A daily build job introduces new questions about how big the images
> are and how many of them we keep, but let's focus on whether the
> change in policy is something we want to adopt before we consider
> those questions.

http://tarballs.openstack.org/kolla/images/ we are already doing this
for last few months. Only difference is that it's hacky and we want
something that's not hacky.

Let's separate resource constrains for now please, because from
current standpoint all the resources we need is a single vm that's
gonna run 1hr every day and some uplink megabytes (probably less than
1gig every day as Docker will cache a lot). If that's an issue, we can
work on it and limit amount of pushes to just version changes,
something we were discussing anyway.

>
>> > Given our security and stable team resources, I'm not entirely
>> > comfortable with us publishing these images, and giving the appearance
>> > that the community *as a whole* is committing to supporting them.
>> > I don't have any objection to someone from the community publishing
>> > them, as long as it is made clear who the 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700:
> For starters, I want to emphasize that fresh set of dockerhub images
> was one of most requested features from Kolla on this summit and few
> other features more or less requires readily-available docker
> registry. Features like full release upgrade gates.
> 
> This will have numerous benefits for users that doesn't have resources
> to put sophisticated CI/staging env, which, I'm willing to bet, is
> still quite significant user base. If we do it correctly (and we will
> do it correctly), images we'll going to push will go through series of
> gates which we have in Kolla (and will have more). So when you pull
> image, you know that it was successfully deployed within scenerios
> available in our gates, maybe even upgrade and increase scenerio
> coverage later? That is a huge benefit for actual users.

I have no doubt that consumers of the images would like us to keep
creating them. We had lots of discussions last week about resource
constraints and sustainable practices, though, and this strikes me
as an area where we're deviating from our history in a way that
will require more maintenance work upstream.

> On 15 May 2017 at 10:34, Doug Hellmann  wrote:
> > Last week at the Forum we had a couple of discussions about
> > collaboration between the various teams building or consuming
> > container images. One topic that came up was deciding how to publish
> > images from the various teams to docker hub or other container
> > registries. While the technical bits seem easy enough to work out,
> > there is still the question of precedence and whether it's a good
> > idea to do so at all.
> >
> > In the past, we have refrained from publishing binary packages in
> > other formats such as debs and RPMs. (We did publish debs way back
> > in the beginning, for testing IIRC, but switched away from them to
> > sdists to be more inclusive.) Since then, we have said it is the
> > responsibility of downstream consumers to build production packages,
> > either as distributors or as a deployer that is rolling their own.
> > We do package sdists for python libraries, push some JavaScript to
> > the NPM registries, and have tarballs of those and a bunch of other
> > artifacts that we build out of our release tools.  But none of those
> > is declared as "production ready," and so the community is not
> > sending the signal that we are responsible for maintaining them in
> > the context of production deployments, beyond continuing to produce
> > new releases when there are bugs.
> 
> So for us that would mean something really hacky and bad. We are
> community driven not company driven project. We don't have Red Hat or
> Canonical teams behind us (we have contributors, but that's
> different).

Although I work at Red Hat, I want to make sure it's clear that my
objection is purely related to community concerns. For this
conversation, I'm wearing my upstream TC and Release team hats.

> > Container images introduce some extra complexity, over the basic
> > operating system style packages mentioned above. Due to the way
> > they are constructed, they are likely to include content we don't
> > produce ourselves (either in the form of base layers or via including
> > build tools or other things needed when assembling the full image).
> > That extra content means there would need to be more tracking of
> > upstream issues (bugs, CVEs, etc.) to ensure the images are updated
> > as needed.
> 
> We can do this by building daily, which was the plan in fact. If we
> build every day you have at most 24hrs old packages, CVEs and things
> like that on non-openstack packages are still maintained by distro
> maintainers.

A daily build job introduces new questions about how big the images
are and how many of them we keep, but let's focus on whether the
change in policy is something we want to adopt before we consider
those questions.

> > Given our security and stable team resources, I'm not entirely
> > comfortable with us publishing these images, and giving the appearance
> > that the community *as a whole* is committing to supporting them.
> > I don't have any objection to someone from the community publishing
> > them, as long as it is made clear who the actual owner is. I'm not
> > sure how easy it is to make that distinction if we publish them
> > through infra jobs, so that may mean some outside process. I also
> > don't think there would be any problem in building images on our
> > infrastructure for our own gate jobs, as long as they are just for
> > testing and we don't push those to any other registries.
> 
> Today we use Kolla account for that and I'm more than happy to keep it
> this way. We license our code with ASL which gives no guarantees.
> Containers will be licensed this way too, so they're available as-is
> and "production readiness" should be decided by everyone who runs it.
> That being said what we *can* promise is that 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 15 May 2017 at 11:47, Sean Dague  wrote:
> On 05/15/2017 01:52 PM, Michał Jastrzębski wrote:
>> For starters, I want to emphasize that fresh set of dockerhub images
>> was one of most requested features from Kolla on this summit and few
>> other features more or less requires readily-available docker
>> registry. Features like full release upgrade gates.
>>
>> This will have numerous benefits for users that doesn't have resources
>> to put sophisticated CI/staging env, which, I'm willing to bet, is
>> still quite significant user base. If we do it correctly (and we will
>> do it correctly), images we'll going to push will go through series of
>> gates which we have in Kolla (and will have more). So when you pull
>> image, you know that it was successfully deployed within scenerios
>> available in our gates, maybe even upgrade and increase scenerio
>> coverage later? That is a huge benefit for actual users.
>
> That concerns me quite a bit. Given the nature of the patch story on
> containers (which is a rebuild), I really feel like users should have
> their own build / CI pipeline locally to be deploying this way. Making
> that easy for them to do, is great, but skipping that required local
> infrastructure puts them in a bad position should something go wrong.

I totally agree they should. Even if they do, it's still would be
additive to gating that we run, so it's even better.

> I do get that many folks want that, but I think it builds in a set of
> expectations that it's not possible to actually meet from an upstream
> perspective.
>
>> On 15 May 2017 at 10:34, Doug Hellmann  wrote:
>>> Last week at the Forum we had a couple of discussions about
>>> collaboration between the various teams building or consuming
>>> container images. One topic that came up was deciding how to publish
>>> images from the various teams to docker hub or other container
>>> registries. While the technical bits seem easy enough to work out,
>>> there is still the question of precedence and whether it's a good
>>> idea to do so at all.
>>>
>>> In the past, we have refrained from publishing binary packages in
>>> other formats such as debs and RPMs. (We did publish debs way back
>>> in the beginning, for testing IIRC, but switched away from them to
>>> sdists to be more inclusive.) Since then, we have said it is the
>>> responsibility of downstream consumers to build production packages,
>>> either as distributors or as a deployer that is rolling their own.
>>> We do package sdists for python libraries, push some JavaScript to
>>> the NPM registries, and have tarballs of those and a bunch of other
>>> artifacts that we build out of our release tools.  But none of those
>>> is declared as "production ready," and so the community is not
>>> sending the signal that we are responsible for maintaining them in
>>> the context of production deployments, beyond continuing to produce
>>> new releases when there are bugs.
>>
>> So for us that would mean something really hacky and bad. We are
>> community driven not company driven project. We don't have Red Hat or
>> Canonical teams behind us (we have contributors, but that's
>> different).
>>
>>> Container images introduce some extra complexity, over the basic
>>> operating system style packages mentioned above. Due to the way
>>> they are constructed, they are likely to include content we don't
>>> produce ourselves (either in the form of base layers or via including
>>> build tools or other things needed when assembling the full image).
>>> That extra content means there would need to be more tracking of
>>> upstream issues (bugs, CVEs, etc.) to ensure the images are updated
>>> as needed.
>>
>> We can do this by building daily, which was the plan in fact. If we
>> build every day you have at most 24hrs old packages, CVEs and things
>> like that on non-openstack packages are still maintained by distro
>> maintainers.
>
> There have been many instances where 24 hours wasn't good enough as
> embargoes end up pretty weird in terms of when things hit mirrors. It
> also assumes that when a CVE hits some other part of the gate or
> infrastructure isn't wedged so that it's not possible to build new
> packages. Or the capacity demands happen during a feature freeze, with
> tons of delay in there. There are many single points of failure in this
> process.
>
>>> Given our security and stable team resources, I'm not entirely
>>> comfortable with us publishing these images, and giving the appearance
>>> that the community *as a whole* is committing to supporting them.
>>> I don't have any objection to someone from the community publishing
>>> them, as long as it is made clear who the actual owner is. I'm not
>>> sure how easy it is to make that distinction if we publish them
>>> through infra jobs, so that may mean some outside process. I also
>>> don't think there would be any problem in building images on our
>>> infrastructure for our own gate jobs, as 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

On 15 May 2017 at 11:19, Davanum Srinivas  wrote:
> Sorry for the top post, Michal, Can you please clarify a couple of things:
>
> 1) Can folks install just one or two services for their specific scenario?

Yes, that's more of a kolla-ansible feature and require a little bit
of ansible know-how, but entirely possible. Kolla-k8s is built to
allow maximum flexibility in that space.

> 2) Can the container images from kolla be run on bare docker daemon?

Yes, but they need to either override our default CMD (kolla_start) or
provide ENVs requred by it, not a huge deal

> 3) Can someone take the kolla container images from say dockerhub and
> use it without the Kolla framework?

Yes, there is no such thing as kolla framework really. Our images
follow stable ABI and they can be deployed by any deploy mechanism
that will follow it. We have several users who wrote their own deploy
mechanism from scratch.

Containers are just blobs with binaries in it. Little things that we
add are kolla_start script to allow our config file management and
some custom startup scripts for things like mariadb to help with
bootstrapping, both are entirely optional.

>
> Thanks,
> Dims
>
> On Mon, May 15, 2017 at 1:52 PM, Michał Jastrzębski  wrote:
>> For starters, I want to emphasize that fresh set of dockerhub images
>> was one of most requested features from Kolla on this summit and few
>> other features more or less requires readily-available docker
>> registry. Features like full release upgrade gates.
>>
>> This will have numerous benefits for users that doesn't have resources
>> to put sophisticated CI/staging env, which, I'm willing to bet, is
>> still quite significant user base. If we do it correctly (and we will
>> do it correctly), images we'll going to push will go through series of
>> gates which we have in Kolla (and will have more). So when you pull
>> image, you know that it was successfully deployed within scenerios
>> available in our gates, maybe even upgrade and increase scenerio
>> coverage later? That is a huge benefit for actual users.
>>
>> On 15 May 2017 at 10:34, Doug Hellmann  wrote:
>>> Last week at the Forum we had a couple of discussions about
>>> collaboration between the various teams building or consuming
>>> container images. One topic that came up was deciding how to publish
>>> images from the various teams to docker hub or other container
>>> registries. While the technical bits seem easy enough to work out,
>>> there is still the question of precedence and whether it's a good
>>> idea to do so at all.
>>>
>>> In the past, we have refrained from publishing binary packages in
>>> other formats such as debs and RPMs. (We did publish debs way back
>>> in the beginning, for testing IIRC, but switched away from them to
>>> sdists to be more inclusive.) Since then, we have said it is the
>>> responsibility of downstream consumers to build production packages,
>>> either as distributors or as a deployer that is rolling their own.
>>> We do package sdists for python libraries, push some JavaScript to
>>> the NPM registries, and have tarballs of those and a bunch of other
>>> artifacts that we build out of our release tools.  But none of those
>>> is declared as "production ready," and so the community is not
>>> sending the signal that we are responsible for maintaining them in
>>> the context of production deployments, beyond continuing to produce
>>> new releases when there are bugs.
>>
>> So for us that would mean something really hacky and bad. We are
>> community driven not company driven project. We don't have Red Hat or
>> Canonical teams behind us (we have contributors, but that's
>> different).
>>
>>> Container images introduce some extra complexity, over the basic
>>> operating system style packages mentioned above. Due to the way
>>> they are constructed, they are likely to include content we don't
>>> produce ourselves (either in the form of base layers or via including
>>> build tools or other things needed when assembling the full image).
>>> That extra content means there would need to be more tracking of
>>> upstream issues (bugs, CVEs, etc.) to ensure the images are updated
>>> as needed.
>>
>> We can do this by building daily, which was the plan in fact. If we
>> build every day you have at most 24hrs old packages, CVEs and things
>> like that on non-openstack packages are still maintained by distro
>> maintainers.
>>
>>> Given our security and stable team resources, I'm not entirely
>>> comfortable with us publishing these images, and giving the appearance
>>> that the community *as a whole* is committing to supporting them.
>>> I don't have any objection to someone from the community publishing
>>> them, as long as it is made clear who the actual owner is. I'm not
>>> sure how easy it is to make that distinction if we publish them
>>> through infra jobs, so that may mean some outside process. I also
>>> don't think there would 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Sean Dague]

On 05/15/2017 01:52 PM, Michał Jastrzębski wrote:
> For starters, I want to emphasize that fresh set of dockerhub images
> was one of most requested features from Kolla on this summit and few
> other features more or less requires readily-available docker
> registry. Features like full release upgrade gates.
> 
> This will have numerous benefits for users that doesn't have resources
> to put sophisticated CI/staging env, which, I'm willing to bet, is
> still quite significant user base. If we do it correctly (and we will
> do it correctly), images we'll going to push will go through series of
> gates which we have in Kolla (and will have more). So when you pull
> image, you know that it was successfully deployed within scenerios
> available in our gates, maybe even upgrade and increase scenerio
> coverage later? That is a huge benefit for actual users.

That concerns me quite a bit. Given the nature of the patch story on
containers (which is a rebuild), I really feel like users should have
their own build / CI pipeline locally to be deploying this way. Making
that easy for them to do, is great, but skipping that required local
infrastructure puts them in a bad position should something go wrong.

I do get that many folks want that, but I think it builds in a set of
expectations that it's not possible to actually meet from an upstream
perspective.

> On 15 May 2017 at 10:34, Doug Hellmann  wrote:
>> Last week at the Forum we had a couple of discussions about
>> collaboration between the various teams building or consuming
>> container images. One topic that came up was deciding how to publish
>> images from the various teams to docker hub or other container
>> registries. While the technical bits seem easy enough to work out,
>> there is still the question of precedence and whether it's a good
>> idea to do so at all.
>>
>> In the past, we have refrained from publishing binary packages in
>> other formats such as debs and RPMs. (We did publish debs way back
>> in the beginning, for testing IIRC, but switched away from them to
>> sdists to be more inclusive.) Since then, we have said it is the
>> responsibility of downstream consumers to build production packages,
>> either as distributors or as a deployer that is rolling their own.
>> We do package sdists for python libraries, push some JavaScript to
>> the NPM registries, and have tarballs of those and a bunch of other
>> artifacts that we build out of our release tools.  But none of those
>> is declared as "production ready," and so the community is not
>> sending the signal that we are responsible for maintaining them in
>> the context of production deployments, beyond continuing to produce
>> new releases when there are bugs.
> 
> So for us that would mean something really hacky and bad. We are
> community driven not company driven project. We don't have Red Hat or
> Canonical teams behind us (we have contributors, but that's
> different).
> 
>> Container images introduce some extra complexity, over the basic
>> operating system style packages mentioned above. Due to the way
>> they are constructed, they are likely to include content we don't
>> produce ourselves (either in the form of base layers or via including
>> build tools or other things needed when assembling the full image).
>> That extra content means there would need to be more tracking of
>> upstream issues (bugs, CVEs, etc.) to ensure the images are updated
>> as needed.
> 
> We can do this by building daily, which was the plan in fact. If we
> build every day you have at most 24hrs old packages, CVEs and things
> like that on non-openstack packages are still maintained by distro
> maintainers.

There have been many instances where 24 hours wasn't good enough as
embargoes end up pretty weird in terms of when things hit mirrors. It
also assumes that when a CVE hits some other part of the gate or
infrastructure isn't wedged so that it's not possible to build new
packages. Or the capacity demands happen during a feature freeze, with
tons of delay in there. There are many single points of failure in this
process.

>> Given our security and stable team resources, I'm not entirely
>> comfortable with us publishing these images, and giving the appearance
>> that the community *as a whole* is committing to supporting them.
>> I don't have any objection to someone from the community publishing
>> them, as long as it is made clear who the actual owner is. I'm not
>> sure how easy it is to make that distinction if we publish them
>> through infra jobs, so that may mean some outside process. I also
>> don't think there would be any problem in building images on our
>> infrastructure for our own gate jobs, as long as they are just for
>> testing and we don't push those to any other registries.
> 
> Today we use Kolla account for that and I'm more than happy to keep it
> this way. We license our code with ASL which gives no guarantees.
> Containers will be licensed this way too, 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Davanum Srinivas]

Sorry for the top post, Michal, Can you please clarify a couple of things:

1) Can folks install just one or two services for their specific scenario?
2) Can the container images from kolla be run on bare docker daemon?
3) Can someone take the kolla container images from say dockerhub and
use it without the Kolla framework?

Thanks,
Dims

On Mon, May 15, 2017 at 1:52 PM, Michał Jastrzębski  wrote:
> For starters, I want to emphasize that fresh set of dockerhub images
> was one of most requested features from Kolla on this summit and few
> other features more or less requires readily-available docker
> registry. Features like full release upgrade gates.
>
> This will have numerous benefits for users that doesn't have resources
> to put sophisticated CI/staging env, which, I'm willing to bet, is
> still quite significant user base. If we do it correctly (and we will
> do it correctly), images we'll going to push will go through series of
> gates which we have in Kolla (and will have more). So when you pull
> image, you know that it was successfully deployed within scenerios
> available in our gates, maybe even upgrade and increase scenerio
> coverage later? That is a huge benefit for actual users.
>
> On 15 May 2017 at 10:34, Doug Hellmann  wrote:
>> Last week at the Forum we had a couple of discussions about
>> collaboration between the various teams building or consuming
>> container images. One topic that came up was deciding how to publish
>> images from the various teams to docker hub or other container
>> registries. While the technical bits seem easy enough to work out,
>> there is still the question of precedence and whether it's a good
>> idea to do so at all.
>>
>> In the past, we have refrained from publishing binary packages in
>> other formats such as debs and RPMs. (We did publish debs way back
>> in the beginning, for testing IIRC, but switched away from them to
>> sdists to be more inclusive.) Since then, we have said it is the
>> responsibility of downstream consumers to build production packages,
>> either as distributors or as a deployer that is rolling their own.
>> We do package sdists for python libraries, push some JavaScript to
>> the NPM registries, and have tarballs of those and a bunch of other
>> artifacts that we build out of our release tools.  But none of those
>> is declared as "production ready," and so the community is not
>> sending the signal that we are responsible for maintaining them in
>> the context of production deployments, beyond continuing to produce
>> new releases when there are bugs.
>
> So for us that would mean something really hacky and bad. We are
> community driven not company driven project. We don't have Red Hat or
> Canonical teams behind us (we have contributors, but that's
> different).
>
>> Container images introduce some extra complexity, over the basic
>> operating system style packages mentioned above. Due to the way
>> they are constructed, they are likely to include content we don't
>> produce ourselves (either in the form of base layers or via including
>> build tools or other things needed when assembling the full image).
>> That extra content means there would need to be more tracking of
>> upstream issues (bugs, CVEs, etc.) to ensure the images are updated
>> as needed.
>
> We can do this by building daily, which was the plan in fact. If we
> build every day you have at most 24hrs old packages, CVEs and things
> like that on non-openstack packages are still maintained by distro
> maintainers.
>
>> Given our security and stable team resources, I'm not entirely
>> comfortable with us publishing these images, and giving the appearance
>> that the community *as a whole* is committing to supporting them.
>> I don't have any objection to someone from the community publishing
>> them, as long as it is made clear who the actual owner is. I'm not
>> sure how easy it is to make that distinction if we publish them
>> through infra jobs, so that may mean some outside process. I also
>> don't think there would be any problem in building images on our
>> infrastructure for our own gate jobs, as long as they are just for
>> testing and we don't push those to any other registries.
>
> Today we use Kolla account for that and I'm more than happy to keep it
> this way. We license our code with ASL which gives no guarantees.
> Containers will be licensed this way too, so they're available as-is
> and "production readiness" should be decided by everyone who runs it.
> That being said what we *can* promise is that our containers passed
> through more or less rigorous gates and that's more than most of
> packages/self-built containers ever do. I think that value would be
> appreciated by small to mid companies that just want to work with
> openstack and don't have means to spare teams/resources for CI.
>
>> I'm raising the issue here to get some more input into how to
>> proceed. Do other people think this concern is 

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Michał Jastrzębski]

For starters, I want to emphasize that fresh set of dockerhub images
was one of most requested features from Kolla on this summit and few
other features more or less requires readily-available docker
registry. Features like full release upgrade gates.

This will have numerous benefits for users that doesn't have resources
to put sophisticated CI/staging env, which, I'm willing to bet, is
still quite significant user base. If we do it correctly (and we will
do it correctly), images we'll going to push will go through series of
gates which we have in Kolla (and will have more). So when you pull
image, you know that it was successfully deployed within scenerios
available in our gates, maybe even upgrade and increase scenerio
coverage later? That is a huge benefit for actual users.

On 15 May 2017 at 10:34, Doug Hellmann  wrote:
> Last week at the Forum we had a couple of discussions about
> collaboration between the various teams building or consuming
> container images. One topic that came up was deciding how to publish
> images from the various teams to docker hub or other container
> registries. While the technical bits seem easy enough to work out,
> there is still the question of precedence and whether it's a good
> idea to do so at all.
>
> In the past, we have refrained from publishing binary packages in
> other formats such as debs and RPMs. (We did publish debs way back
> in the beginning, for testing IIRC, but switched away from them to
> sdists to be more inclusive.) Since then, we have said it is the
> responsibility of downstream consumers to build production packages,
> either as distributors or as a deployer that is rolling their own.
> We do package sdists for python libraries, push some JavaScript to
> the NPM registries, and have tarballs of those and a bunch of other
> artifacts that we build out of our release tools.  But none of those
> is declared as "production ready," and so the community is not
> sending the signal that we are responsible for maintaining them in
> the context of production deployments, beyond continuing to produce
> new releases when there are bugs.

So for us that would mean something really hacky and bad. We are
community driven not company driven project. We don't have Red Hat or
Canonical teams behind us (we have contributors, but that's
different).

> Container images introduce some extra complexity, over the basic
> operating system style packages mentioned above. Due to the way
> they are constructed, they are likely to include content we don't
> produce ourselves (either in the form of base layers or via including
> build tools or other things needed when assembling the full image).
> That extra content means there would need to be more tracking of
> upstream issues (bugs, CVEs, etc.) to ensure the images are updated
> as needed.

We can do this by building daily, which was the plan in fact. If we
build every day you have at most 24hrs old packages, CVEs and things
like that on non-openstack packages are still maintained by distro
maintainers.

> Given our security and stable team resources, I'm not entirely
> comfortable with us publishing these images, and giving the appearance
> that the community *as a whole* is committing to supporting them.
> I don't have any objection to someone from the community publishing
> them, as long as it is made clear who the actual owner is. I'm not
> sure how easy it is to make that distinction if we publish them
> through infra jobs, so that may mean some outside process. I also
> don't think there would be any problem in building images on our
> infrastructure for our own gate jobs, as long as they are just for
> testing and we don't push those to any other registries.

Today we use Kolla account for that and I'm more than happy to keep it
this way. We license our code with ASL which gives no guarantees.
Containers will be licensed this way too, so they're available as-is
and "production readiness" should be decided by everyone who runs it.
That being said what we *can* promise is that our containers passed
through more or less rigorous gates and that's more than most of
packages/self-built containers ever do. I think that value would be
appreciated by small to mid companies that just want to work with
openstack and don't have means to spare teams/resources for CI.

> I'm raising the issue here to get some more input into how to
> proceed. Do other people think this concern is overblown? Can we
> mitigate the risk by communicating through metadata for the images?
> Should we stick to publishing build instructions (Dockerfiles, or
> whatever) instead of binary images? Are there other options I haven't
> mentioned?

Today we do publish build instructions, that's what Kolla is. We also
publish built containers already, just we do it manually on release
today. If we decide to block it, I assume we should stop doing that
too? That will hurt users who uses this piece of Kolla, and I'd hate
to hurt our 

[openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Doug Hellmann]

Last week at the Forum we had a couple of discussions about
collaboration between the various teams building or consuming
container images. One topic that came up was deciding how to publish
images from the various teams to docker hub or other container
registries. While the technical bits seem easy enough to work out,
there is still the question of precedence and whether it's a good
idea to do so at all.

In the past, we have refrained from publishing binary packages in
other formats such as debs and RPMs. (We did publish debs way back
in the beginning, for testing IIRC, but switched away from them to
sdists to be more inclusive.) Since then, we have said it is the
responsibility of downstream consumers to build production packages,
either as distributors or as a deployer that is rolling their own.
We do package sdists for python libraries, push some JavaScript to
the NPM registries, and have tarballs of those and a bunch of other
artifacts that we build out of our release tools.  But none of those
is declared as "production ready," and so the community is not
sending the signal that we are responsible for maintaining them in
the context of production deployments, beyond continuing to produce
new releases when there are bugs.

Container images introduce some extra complexity, over the basic
operating system style packages mentioned above. Due to the way
they are constructed, they are likely to include content we don't
produce ourselves (either in the form of base layers or via including
build tools or other things needed when assembling the full image).
That extra content means there would need to be more tracking of
upstream issues (bugs, CVEs, etc.) to ensure the images are updated
as needed.

Given our security and stable team resources, I'm not entirely
comfortable with us publishing these images, and giving the appearance
that the community *as a whole* is committing to supporting them.
I don't have any objection to someone from the community publishing
them, as long as it is made clear who the actual owner is. I'm not
sure how easy it is to make that distinction if we publish them
through infra jobs, so that may mean some outside process. I also
don't think there would be any problem in building images on our
infrastructure for our own gate jobs, as long as they are just for
testing and we don't push those to any other registries.

I'm raising the issue here to get some more input into how to
proceed. Do other people think this concern is overblown? Can we
mitigate the risk by communicating through metadata for the images?
Should we stick to publishing build instructions (Dockerfiles, or
whatever) instead of binary images? Are there other options I haven't
mentioned?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev