Re: [openstack-dev] [Fuel] Packaging CI for Fuel

[Jeremy Stanley]

On 2016-03-22 15:20:22 + (+), Tristan Cacqueray wrote:
[...]
> With only my VMT hat on, this makes me wonder why the packaging needs
> special care. Is there a reason why stable branch aren't built continuously?
[...]

My concern was more over handling packaging-specific security
vulnerabilities (loose file permissions, risky default
configuration, that sort of thing) which is what I think
"vulnerability management" of packages would entail.

But also all generated packages published from the same suite would
need consistent support coverage not just for the packaging itself
but also for all the software being packaged--sysadmins consuming
our packages aren't going to take the time to figure out which of
packages installed from there are "supported" by us and which
aren't.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Packaging CI for Fuel

[Tristan Cacqueray]

On 03/19/2016 06:53 PM, Jeremy Stanley wrote:
> On 2016-03-19 05:10:18 -0500 (-0500), Monty Taylor wrote:
> [...]
>> It would also be good to tie off with the security team about
>> this. One of the reasons we stopped publishing debs years ago is
>> that it made us a de-facto derivative distro. People were using
>> our packages in production, including backports we'd built in
>> support of those packages, but our backports were not receiving
>> security/CVE attention, so we were concerned that we were causing
>> people to be exposed to issues. Of course. "we" was thierry,
>> soren, jeblair and I, which is clearly not enough people. Now we
>> have a whole security team and people who DO track CVEs - so if
>> they're willing to at least keep an eye on things we publish in a
>> repo, then I think we're in good shape to publish a repo with
>> backports in it.
> [...]
> 
> Please be aware that the VMT's direct support for triaging, tracking
> and announcing vulnerabilities/fixes only extends to a very small
> subset of OpenStack already. With both my VMT and Infra hats on, I
> really don't feel like we have either the workforce nor expertise to
> make security guarantees about our auto-built packages. We'll make a
> best effort attempt to rebuild packages as soon as possible after
> patches merge to their corresponding repos, assuming the toolchain
> and our CI are having a good day.
> 

With only my VMT hat on, this makes me wonder why the packaging needs
special care. Is there a reason why stable branch aren't built continuously?

Otherwise I agree with Jeremy, VMT is already quite busy supporting
vulnerability:managed projects' master branch along with supported
stable branch. Adding more branches to track doesn't seem like the right
approach.

-Tristan

> I'm not against building and publishing packages, but we need to
> make big ugly disclaimers everywhere we can that these are not
> security supported by us, not intended for production use, and if
> they break your deployment you get to keep all the pieces. Users of
> legitimate distros need to consider those packages superior to ours
> in every way, since I really don't want to be on the hook to support
> them for more than validation purposes.
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Packaging CI for Fuel

[Thomas Goirand]

On 03/19/2016 11:10 AM, Monty Taylor wrote:
> The patch looks good, but it conflicts with the
> move-nova-jobs-to-db-macro and the "add debian jessie support for bindep
> fallback" change. Rather than fighting the rebase fight, let's put a
> brief hold on this (sorry, I know) and land it as soon as those land.

Ok.

It'd be nice if someone could ping me when I can resume my work on it,
if you guys are following closely the other 2 patches (if not, I'll try
to remember to check for them).

> We'll need to work to make sure that we're using zuul-cloner to get the
> right things checked out

Is this a macro the package build needs to use?

> Luckily, we've got and apt-repository infrastructure already set up and
> ready in infra thanks to the mirror work we did this last cycle. It's
> using reprepro fwiw.

Outch ! I had multiple very bad experiences with reprepro. For example,
if we rebuild the same package with the same version, reprepro will
*not* pick-up the package. And we do need to do this, because the
debian/changelog needs to match what is uploaded to the Debian archive,
and it cannot increment.

The thing is, maintaining a Debian repository can be done with a very
small shell script like this one:

http://anonscm.debian.org/cgit/openstack/openstack-pkg-tools.git/tree/build-tools/pkgos-scan-repo

Hopefully, we can switch to something like this that allows more control
than reprepro allows.

> I believe we should add jessie to the list of things we mirror in it,
> and then also add a volume to hold things we publish ourselves.

These are typically "one off" backports which we don't need to care care
much of, but which are needed for other OpenStack to build or run.
Probably something based on a yaml file listing all the packages could
be enough.

> We'll also need to move from having an unsigned reprepro to a signed
> reprepro if we're going to publish our own packages. We've not been
> signing the repo so far because we've sort of wanted to discourage use
> of our mirror outside of the gate - but it turns  out our mirror is
> AMAZING - so I think it's time we change that.

Ok. The script I listed above signs the repo, it's not hard to do, as
you probably know already.

>> Finally, we'll need a way to build backports from Sid and also publish
>> them.
> 
> Hrm. We might want to mirror sid then too. I'd like to talk about the
> backport building process - hopefully a process that does not need to
> require us making a repo in gerrit for each package we want to backport
> and include in our repo.

Exactly!

The list of packages, you may find it here:
http://mitaka-jessie.pkgs.mirantis.com/debian/pool/jessie-mitaka-backports-nochange/

Or, more easily, in the Sources file here:
ftp://mitaka-jessie.pkgs.mirantis.com/debian/dists/jessie-mitaka-backports-nochange/main/source/

The list of packages could be maintained in a .yaml file for example,
then parsed to regularly maintain the backports, and sending an alert if
one package fails to build.

> It would also be good to tie off with the security team about this. One
> of the reasons we stopped publishing debs years ago is that it made us a
> de-facto derivative distro. People were using our packages in
> production, including backports we'd built in support of those packages,
> but our backports were not receiving security/CVE attention, so we were
> concerned that we were causing people to be exposed to issues.

I'd like to also take care of these packages, and upload them to Sid. I
already packaged some of them, though I was stopped because of the lack
of support for statsd >= 3.x (which is in Sid), which has since been
added. I could resume that work once Mitaka is done (in the mean while,
I'm *very* busy with it).

> We'll also want to make sure we're building packages for trusty and xenial.

All I write works for both. It is fully my intention to support Trusty
and Xenial as well as Jessie and hopefully Sid (with probably Sid as
non-voting, as it will break too often).

> Yay for movement!

+1 !!!
If you're really going to be the PTL *and* help this to happen, that's
fantastic. Thanks for your comments already.

Cheers,

Thomas Goirand (zigo)

P.S: ACK for Fungi's reply about security.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] Packaging CI for Fuel

[Aleksandra Fedorova]

Hi, everyone,

we'd like to announce Packaging CI for Fuel.

For now there are no extended tests, just package builds.
Package-based test will be added later in Newton development cycle.

Current workflow:

* for every request we build a package and publish it to temporary
repository, which you can use in local tests

http://packages.fuel-infra.org/review/

* on merge we build package and publish to main repo at:

http://packages.fuel-infra.org/repositories/

Links and description is available at:

https://wiki.openstack.org/wiki/Fuel/CI#Packaging_CI_Overview

We plan to enable voting for this CI by the end of the week. It will
vote as "Fuel Packaging CI".

-- 
Aleksandra Fedorova
CI Team Lead
bookwar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Packaging CI for Fuel

[Vladimir Kozhukalov]

Hi,

> Are there any effort on OpenStack packaging?

The short answer is yes. We are putting efforts to migrate all
our packaging activities to the community RPM/DEB projects.

Long story is as follows:

At the moment Fuel is distributed as a set of RPM packages. This
Packaging CI that Aleksandra mentioned is nothing more
but a copy of CI that we have been using at Mirantis for about
two years. At the moment we are working hard to split Fuel
into upstream (community) and downstream (part of MOS) and this
CI is a part of this work.

Currently, OpenStack RPM project is at the early development stage.
It is another project (not Fuel) and I hope it will be finally
possible to build RPM packages using OpenStack Infrastructure.
We (Fuel) are in contact with OpenStack RPM team and we are planning
to move all Fuel RPM specs under this project. I hope this migration
will be finished in Newton cycle.

Besides, packaging is not just preparing RPM/DEB specs. We also
need tools to build packages as well as CI strategy
 - get spec here
 - get source code there
 - prepare build environment
 - build packages
 - publish packages to testing repository
 - test packages
 - publish packages to current public repository
Fuel Packaging CI already does this. And the fact that we made a public
Packaging CI instance reflects our intention to share our experience.

I'd also like to mention our recent initiative Packetary [1] which is
a tool that is to cover the whole RPM/DEB packaging domain
(building packages, building repositories, clonning repositories).
It is not something totally new, it is to become a single convenient API to
widely used tools like createrepo, python-debian, mock, sbuild, etc.
This tools could be potentially used as a part of whatever CI or by
a regular user via CLI.

As for DEB packaging, as Thomas wrote, he is currently working on making
this possible to build DEB packages (including Fuel) using OpenStack
Infrastructure.

[1] https://wiki.openstack.org/wiki/Packetary


Vladimir Kozhukalov

On Fri, Mar 18, 2016 at 12:11 AM, Thomas Goirand  wrote:

> On 03/16/2016 06:22 PM, Emilien Macchi wrote:
> > Are they any effort on OpenStack packaging?
> >
> > http://governance.openstack.org/reference/projects/packaging-deb.html
> > http://governance.openstack.org/reference/projects/packaging-rpm.html
> >
> > I would like to see packaging built & tested by OpenStack Infra, so
> > downstream CI (Fuel, Puppet OpenStack, TripleO, Kolla, etc) could use
> > it as a single place and efforts would converge.
>
> Hi Emilien,
>
> As you know, things were a bit stuck, with the Debian image patch not
> being approved. But it has changed, and we do have a debian-jessie image
> in infra now. Therefore, I've move to the next step, which is actually
> building packages. Here's the CR:
>
> https://review.openstack.org/#/c/294022/
>
> I've been able to test the pkgdeb-install-sbuild.sh script that I'm
> proposing to setup sbuild on a copy of the Debian image (thanks a lot to
> Pabelanger and Fungi copy the image, and give it to me for download),
> and sbuild was setup properly. The pkgdeb-build-pkg.sh also worked,
> though I'm not 100% sure yet about the content of
> /home/jenkins/workspace/${JOB_NAME}, if it will have the correct branch
> or what, but everything else should be working to build packages.
>
> Once packages are built, then we will want to publish them somewhere.
> That's the part where there's lots of unknown. This has so far never
> been done on OpenStack infra. Hopefully, our new PTL will probably help
> here (or someone else from infra)! :) Also, managing a Debian repository
> isn't really hard to do: one can generate the necessary artifacts with a
> small shell script which uses apt-ftparchive (you can look how its done
> at src/pkgos-scan-repo in openstack-pkg-tools).
>
> Finally, we'll need a way to build backports from Sid and also publish
> them.
>
> That's where we are now. Let's go back to the first step, which is the
> CR linked above. Help and comments welcome.
>
> Cheers,
>
> Thomas Goirand (zigo)
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Packaging CI for Fuel

[Jeremy Stanley]

On 2016-03-19 05:10:18 -0500 (-0500), Monty Taylor wrote:
[...]
> It would also be good to tie off with the security team about
> this. One of the reasons we stopped publishing debs years ago is
> that it made us a de-facto derivative distro. People were using
> our packages in production, including backports we'd built in
> support of those packages, but our backports were not receiving
> security/CVE attention, so we were concerned that we were causing
> people to be exposed to issues. Of course. "we" was thierry,
> soren, jeblair and I, which is clearly not enough people. Now we
> have a whole security team and people who DO track CVEs - so if
> they're willing to at least keep an eye on things we publish in a
> repo, then I think we're in good shape to publish a repo with
> backports in it.
[...]

Please be aware that the VMT's direct support for triaging, tracking
and announcing vulnerabilities/fixes only extends to a very small
subset of OpenStack already. With both my VMT and Infra hats on, I
really don't feel like we have either the workforce nor expertise to
make security guarantees about our auto-built packages. We'll make a
best effort attempt to rebuild packages as soon as possible after
patches merge to their corresponding repos, assuming the toolchain
and our CI are having a good day.

I'm not against building and publishing packages, but we need to
make big ugly disclaimers everywhere we can that these are not
security supported by us, not intended for production use, and if
they break your deployment you get to keep all the pieces. Users of
legitimate distros need to consider those packages superior to ours
in every way, since I really don't want to be on the hook to support
them for more than validation purposes.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Packaging CI for Fuel

[Emilien Macchi]

Are they any effort on OpenStack packaging?

http://governance.openstack.org/reference/projects/packaging-deb.html
http://governance.openstack.org/reference/projects/packaging-rpm.html

I would like to see packaging built & tested by OpenStack Infra, so
downstream CI (Fuel, Puppet OpenStack, TripleO, Kolla, etc) could use
it as a single place and efforts would converge.

On Wed, Mar 16, 2016 at 11:56 AM, Aleksandra Fedorova
 wrote:
> Hi, everyone,
>
> we'd like to announce Packaging CI for Fuel.
>
> For now there are no extended tests, just package builds.
> Package-based test will be added later in Newton development cycle.
>
> Current workflow:
>
> * for every request we build a package and publish it to temporary
> repository, which you can use in local tests
>
> http://packages.fuel-infra.org/review/
>
> * on merge we build package and publish to main repo at:
>
> http://packages.fuel-infra.org/repositories/
>
> Links and description is available at:
>
> https://wiki.openstack.org/wiki/Fuel/CI#Packaging_CI_Overview
>
> We plan to enable voting for this CI by the end of the week. It will
> vote as "Fuel Packaging CI".
>
> --
> Aleksandra Fedorova
> CI Team Lead
> bookwar
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Packaging CI for Fuel

[Thomas Goirand]

On 03/16/2016 06:22 PM, Emilien Macchi wrote:
> Are they any effort on OpenStack packaging?
> 
> http://governance.openstack.org/reference/projects/packaging-deb.html
> http://governance.openstack.org/reference/projects/packaging-rpm.html
> 
> I would like to see packaging built & tested by OpenStack Infra, so
> downstream CI (Fuel, Puppet OpenStack, TripleO, Kolla, etc) could use
> it as a single place and efforts would converge.

Hi Emilien,

As you know, things were a bit stuck, with the Debian image patch not
being approved. But it has changed, and we do have a debian-jessie image
in infra now. Therefore, I've move to the next step, which is actually
building packages. Here's the CR:

https://review.openstack.org/#/c/294022/

I've been able to test the pkgdeb-install-sbuild.sh script that I'm
proposing to setup sbuild on a copy of the Debian image (thanks a lot to
Pabelanger and Fungi copy the image, and give it to me for download),
and sbuild was setup properly. The pkgdeb-build-pkg.sh also worked,
though I'm not 100% sure yet about the content of
/home/jenkins/workspace/${JOB_NAME}, if it will have the correct branch
or what, but everything else should be working to build packages.

Once packages are built, then we will want to publish them somewhere.
That's the part where there's lots of unknown. This has so far never
been done on OpenStack infra. Hopefully, our new PTL will probably help
here (or someone else from infra)! :) Also, managing a Debian repository
isn't really hard to do: one can generate the necessary artifacts with a
small shell script which uses apt-ftparchive (you can look how its done
at src/pkgos-scan-repo in openstack-pkg-tools).

Finally, we'll need a way to build backports from Sid and also publish them.

That's where we are now. Let's go back to the first step, which is the
CR linked above. Help and comments welcome.

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Packaging CI for Fuel

[Monty Taylor]

On 03/17/2016 03:11 PM, Thomas Goirand wrote:

On 03/16/2016 06:22 PM, Emilien Macchi wrote:

Are they any effort on OpenStack packaging?

http://governance.openstack.org/reference/projects/packaging-deb.html
http://governance.openstack.org/reference/projects/packaging-rpm.html

I would like to see packaging built & tested by OpenStack Infra, so
downstream CI (Fuel, Puppet OpenStack, TripleO, Kolla, etc) could use
it as a single place and efforts would converge.


Hi Emilien,

As you know, things were a bit stuck, with the Debian image patch not
being approved. But it has changed, and we do have a debian-jessie image
in infra now. Therefore, I've move to the next step, which is actually
building packages. Here's the CR:

https://review.openstack.org/#/c/294022/


Woot!

The patch looks good, but it conflicts with the 
move-nova-jobs-to-db-macro and the "add debian jessie support for bindep 
fallback" change. Rather than fighting the rebase fight, let's put a 
brief hold on this (sorry, I know) and land it as soon as those land.



I've been able to test the pkgdeb-install-sbuild.sh script that I'm
proposing to setup sbuild on a copy of the Debian image (thanks a lot to
Pabelanger and Fungi copy the image, and give it to me for download),
and sbuild was setup properly. The pkgdeb-build-pkg.sh also worked,
though I'm not 100% sure yet about the content of
/home/jenkins/workspace/${JOB_NAME}, if it will have the correct branch
or what, but everything else should be working to build packages.


We'll need to work to make sure that we're using zuul-cloner to get the 
right things checked out - this will be important for making sure that 
Depends-On works (that way someone can propose a nova change, and you 
can propose a packaging change that Depends-On that change and we can 
make sure we have a packaging change to handle it before it even lands)



Once packages are built, then we will want to publish them somewhere.
That's the part where there's lots of unknown. This has so far never
been done on OpenStack infra. Hopefully, our new PTL will probably help
here (or someone else from infra)! :) Also, managing a Debian repository
isn't really hard to do: one can generate the necessary artifacts with a
small shell script which uses apt-ftparchive (you can look how its done
at src/pkgos-scan-repo in openstack-pkg-tools).


Luckily, we've got and apt-repository infrastructure already set up and 
ready in infra thanks to the mirror work we did this last cycle. It's 
using reprepro fwiw.


I believe we should add jessie to the list of things we mirror in it, 
and then also add a volume to hold things we publish ourselves.


We'll also need to move from having an unsigned reprepro to a signed 
reprepro if we're going to publish our own packages. We've not been 
signing the repo so far because we've sort of wanted to discourage use 
of our mirror outside of the gate - but it turns  out our mirror is 
AMAZING - so I think it's time we change that.



Finally, we'll need a way to build backports from Sid and also publish them.


Hrm. We might want to mirror sid then too. I'd like to talk about the 
backport building process - hopefully a process that does not need to 
require us making a repo in gerrit for each package we want to backport 
and include in our repo.


It would also be good to tie off with the security team about this. One 
of the reasons we stopped publishing debs years ago is that it made us a 
de-facto derivative distro. People were using our packages in 
production, including backports we'd built in support of those packages, 
but our backports were not receiving security/CVE attention, so we were 
concerned that we were causing people to be exposed to issues. Of 
course. "we" was thierry, soren, jeblair and I, which is clearly not 
enough people. Now we have a whole security team and people who DO track 
CVEs - so if they're willing to at least keep an eye on things we 
publish in a repo, then I think we're in good shape to publish a repo 
with backports in it.


We'll also want to make sure we're building packages for trusty and xenial.


That's where we are now. Let's go back to the first step, which is the
CR linked above. Help and comments welcome.


Yay for movement!


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev