Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/15/2015 10:43 PM, Paul Belanger wrote: On 06/15/2015 03:03 PM, Allison Randal wrote: On 06/15/2015 11:48 AM, Thomas Goirand wrote: On 06/15/2015 04:55 PM, James Page wrote: The problem of managing delta and allowing a good level of distribution independence is still going to continue to exist and will be more difficult to manage due to the tighter coupling of development process and teams than we have today. On this basis, we're -1 on taking this proposal forward. That said, we do appreciate that the Ubuntu packaging for OpenStack is not as accessible as it might be using Bazaar as a VCS. In order to provide a more familiar experience to developers and operators looking to contribute to the wider Openstack ecosystem we will be moving our OpenStack packaging branches over to the new Git support in Launchpad in the next few weeks. [...] During our discussions at the Summit, you seemed to be enthusiastic about pushing our packaging to Stackforge. Then others told me to push it to the /openstack namespace to make it more big tent-ish, which made me very excited about the idea. So far, I've been very happy of the reboot of our collaboration, and felt like it was just awesome new atmosphere. So I have to admit I'm a bit disappointed to read the above, even though I do understand the reasoning. James is right. This discussion thread put a lot of faith in the possibility that moving packaging efforts under the OpenStack umbrella would magically solve our key blocking issues. (I'm guilty of it as much as anyone else.) But really, we the collaborators are the ones who have to solve those blocking issues, and we'll have to do it together, no matter what banner we do it under. Anyway, does this mean that you don't want to push packaging to /stackforge either, which was the idea we shared at the summit? I'm a bit lost on what I should do now, as what was exciting was enabling operation people to contribute. I'll think about it and see what to do next. It doesn't really matter where the repos are located, we can still collaborate. Just moving Ubuntu's openstack repos to git and the Debian Python Modules Team repos to git will be a massive step forward. While I agree those points are valid, and going to be helpful, moving under OpenStack (even Stackforge) does also offer the chance to get more test integration upstream (not saying this was the original scope). However, this could also be achieved by 3rd party integration too. I'm still driving forward with some -infra specific packaging for Debian / Fedora ATM (zuul packaging). Mostly because of -infra needs for packages. Not saying that is a reason to reconsider, but there is the need for -infra to consume packages from upstream. Thomas where does this leave you (Debian). Are you still considering the move to upstream? Hi Paul, FYI, I tried packaging Zuul and Nodepool for Debian, and saw the work which has been done packaging it for -infra. I do plan to contribute to that soon (as I have already some changes done). So I will at least help for this. As for moving packages to stackforge, I hope to start this effort, yes, but probably not for the packages we share with Ubuntu, as this would be problematic. So I have to start thinking about how to do it without destroying the ongoing collaboration with James Page team. I also need to have meetings with the Mirantis MOS packaging team, which will happen in Moscow at the end of the month. So, maybe, the best way to start would be with Zuul and Nodepool, as you need that. Your contribution is very valuable. I'm not sure how I can help you to help, and how to start... Let's agree to catch up on IRC and discuss that, ok? Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/16/2015 06:41 PM, Allison Randal wrote: On 06/15/2015 01:43 PM, Paul Belanger wrote: While I agree those points are valid, and going to be helpful, moving under OpenStack (even Stackforge) does also offer the chance to get more test integration upstream (not saying this was the original scope). However, this could also be achieved by 3rd party integration too. Nod, 3rd party integration is worth exploring. I'm still driving forward with some -infra specific packaging for Debian / Fedora ATM (zuul packaging). Mostly because of -infra needs for packages. Not saying that is a reason to reconsider, but there is the need for -infra to consume packages from upstream. I suspect that, at least initially, the needs of -infra specific packaging will be quite different than the needs of general-purpose packaging in Debian/Fedora distros. Trying to tightly couple the two will just bog you down in trying to solve far too many problems for far too many people. But, I also suspect that -infra packaging will be quite minimal and intended for the services to be configured by puppet, so there's a very good chance that if you sprint ahead and just do it, your style of packaging will end up feeding back into future packaging in the distros. Allison As I wrote, I intend to contribute to that, and get the resulting packages uploaded to Debian. Currently, there's a few issues about missing dependencies in Debian, which I'm trying to fix first (I don't maintain these packages, and as we have strong package ownership in Debian, I have to get in touch with the maintainer first... and that can take some time!). Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/16/2015 12:41 PM, Allison Randal wrote: On 06/15/2015 01:43 PM, Paul Belanger wrote: While I agree those points are valid, and going to be helpful, moving under OpenStack (even Stackforge) does also offer the chance to get more test integration upstream (not saying this was the original scope). However, this could also be achieved by 3rd party integration too. Nod, 3rd party integration is worth exploring. I'm still driving forward with some -infra specific packaging for Debian / Fedora ATM (zuul packaging). Mostly because of -infra needs for packages. Not saying that is a reason to reconsider, but there is the need for -infra to consume packages from upstream. I suspect that, at least initially, the needs of -infra specific packaging will be quite different than the needs of general-purpose packaging in Debian/Fedora distros. Trying to tightly couple the two will just bog you down in trying to solve far too many problems for far too many people. But, I also suspect that -infra packaging will be quite minimal and intended for the services to be configured by puppet, so there's a very good chance that if you sprint ahead and just do it, your style of packaging will end up feeding back into future packaging in the distros. My thoughts exactly. I believe by the next summit, we should have a base in -infra for producing packages (unsure about consuming ATM). Interesting times ahead. Allison __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/15/2015 01:43 PM, Paul Belanger wrote: While I agree those points are valid, and going to be helpful, moving under OpenStack (even Stackforge) does also offer the chance to get more test integration upstream (not saying this was the original scope). However, this could also be achieved by 3rd party integration too. Nod, 3rd party integration is worth exploring. I'm still driving forward with some -infra specific packaging for Debian / Fedora ATM (zuul packaging). Mostly because of -infra needs for packages. Not saying that is a reason to reconsider, but there is the need for -infra to consume packages from upstream. I suspect that, at least initially, the needs of -infra specific packaging will be quite different than the needs of general-purpose packaging in Debian/Fedora distros. Trying to tightly couple the two will just bog you down in trying to solve far too many problems for far too many people. But, I also suspect that -infra packaging will be quite minimal and intended for the services to be configured by puppet, so there's a very good chance that if you sprint ahead and just do it, your style of packaging will end up feeding back into future packaging in the distros. Allison __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/15/2015 10:55 AM, James Page wrote: We understand and have communicated from the start of this conversation that we will need to be able to maintain deltas between Debian and Ubuntu - for both technical reasons, in the way the distributions work (think Ubuntu main vs universe), as well as objectives that each distribution has in terms of the way packaging should work. Hi James, For the benefit of the TC members (such as myself) that do not have a great background in packaging internals, would you mind describing one or two of the deltas you describe above? I'm really wondering what these things look like and how big the difference is from the Debian packaging recipes (is that the right word, even?) All the best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Thomas On 15/06/15 19:48, Thomas Goirand wrote: [...] During our discussions at the Summit, you seemed to be enthusiastic about pushing our packaging to Stackforge. Then others told me to push it to the /openstack namespace to make it more big tent-ish, which made me very excited about the idea. The idea of having packaging on stackforge seemed appealing; however its always good to reflect on these things after some thought away from the summit, hence my email yesterday. So far, I've been very happy of the reboot of our collaboration, and felt like it was just awesome new atmosphere. So I have to admit I'm a bit disappointed to read the above, even though I do understand the reasoning. Ditto on the reboot of collaboration - I think we've made good steps forward in the last two weeks to work together in the right ways between Ubuntu and Debian. We've still some way to go :-). Anyway, does this mean that you don't want to push packaging to /stackforge either, which was the idea we shared at the summit? Yes. I'm a bit lost on what I should do now, as what was exciting was enabling operation people to contribute. I'll think about it and see what to do next. Lets focus on the seed of collaboration we have going right now and see how that plays out. Cheers James - -- James Page Ubuntu and Debian Developer james.p...@ubuntu.com jamesp...@debian.org -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBCAAGBQJVgAARAAoJEL/srsug59jDtfgQAJgTXuoOKiXGt6HbcGP+VpeM o1SUq8EAg70rpgxtz97wqdW5rZ8u7PRVepx24kFdGSnzvoFMU9Sb3wgM/osjxaig +u2gELObxAKDzKQM/sYWxuIiMNEXwkYWo/PP/hS8WUfsImtmW991ZoQAseD9Yw+k 5D53UxnLbMxdv7Q2qMLEICQwOK6drEQu1Z7GG/X47plwsgVPbHjZ3apsOpbA41zA 8XniBVyAfSurS3vDqCwBk16YcYhsFPn9OjZZ9TDbWTsIuRr6/zBWObWTwwBPqFGv +ks45fhIfLHQJo/BN7MclDQE2Tk7Fc5fTdVme6jUBY8RbIfAw3T8S+LdmtQGL6eP ug7CcDi0aeUIm0s1Z1Cb7HKicoaBrkqPwpIs5TYkAETFiBuh6/qbCDXxPHyR+8/F /r2jFCZmzdyPw2GW/vTkripQb2YmvsvdjkY5tBXQEaFa6bp804Li3wIBXY0DDkzw plFXKAUpyGip/Oj14aHNM7gZZ5dGAjjAJMBLpKcHYoM/B7grup2/ySI/b9w/v3Pj NuKZzEhfEC6W7tuBmJ0F3LyoBTz0uBtY0ar3xIO9yU87v7jz33N+2w6Sei9qaBkh 6Fsh0NpHc5u1NLQ3I+8c3jgRXBTl1+xOPDXZokFnaoM8sl4VdTUx7kp6yD1nrpLO z4tsCjcPXAcLVtxsyW/y =++l/ -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi All
On 27/05/15 09:14, Thomas Goirand wrote:
tl;dr: - We'd like to push distribution packaging of OpenStack on
upstream gerrit with reviews. - The intention is to better share
the workload, and improve the overall QA for packaging *and*
upstream. - The goal is *not* to publish packages upstream -
There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get
started. - There's an ongoing discussion about using a distribution
specific namespace, my own opinion here is that using
/openstack-pkg-{deb,rpm} or /stackforge-pkg-{deb,rpm} would be the
most convenient because of a number of technical reasons like the
amount of Git repository. - Finally, let's not discuss for too long
and let's do it!!!
While working to re-align the dependency chain for OpenStack between
Debian and Ubuntu 15.10, and in preparation for the first Liberty
milestone, my team and I have been reflecting on the proposal to make
Ubuntu and Debian packaging of OpenStack a full OpenStack project.
We’ve come to the conclusion that for Debian and Ubuntu the best place
for us to collaborate on packaging is actually in the distributions,
as we do today, and not upstream in OpenStack.
Ubuntu has collaborated effectively with Debian since its inception
and has an effective set of tools to support the flow of packaging
(from Debian), bug reports and patches (to Debian) that have proven
effective, in terms of both efficiency and value, ever since I’ve been
working across both distributions.
This process allows each distribution to maintain its own direction,
whilst ensuring that any value that might be derived from
collaboration is supported with as minimal overhead as possible.
We understand and have communicated from the start of this
conversation that we will need to be able to maintain deltas between
Debian and Ubuntu - for both technical reasons, in the way the
distributions work (think Ubuntu main vs universe), as well as
objectives that each distribution has in terms of the way packaging
should work.
We don’t think that’s going to be made any easier by moving all of the
packaging under the OpenStack project - it just feels like we’re
trying to push a solved problem somewhere else, and then re-solve it
in a whole new set of ways.
The problem of managing delta and allowing a good level of
distribution independence is still going to continue to exist and will
be more difficult to manage due to the tighter coupling of development
process and teams than we have today.
On this basis, we're -1 on taking this proposal forward.
That said, we do appreciate that the Ubuntu packaging for OpenStack is
not as accessible as it might be using Bazaar as a VCS. In order to
provide a more familiar experience to developers and operators looking
to contribute to the wider Openstack ecosystem we will be moving our
OpenStack packaging branches over to the new Git support in Launchpad
in the next few weeks.
Regards
James
- --
James Page
Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
iQIcBAEBCAAGBQJVfudNAAoJEL/srsug59jD2isQAKtp9gSQ7FC0dy664wkSIqp3
ztmtIGPu5k0kZsg0sSxie3lA6mmRtv0m3sZWweNXfObXKwrWSgJSNynYDOhhiC7u
zlijQwEoY264byd7I+qacCdGBPi8fkXImB+6yx6OdJuHO+DcF/lBhF/5XW+wEwMa
j5GLN/UML+AO/Vp1BNBWholdCy/vm8SYDWtD3952R3fasBusCzpGj/52Pe3JifV6
kYWhnoihSQn+U02SXUc4/JETl/3o94EKp5/eu9We49sEdgHudSF3o6MdyLom2NfM
BNMpWs4iNWz7BlgqoDULotrFORRjQawru9R5StouB+wORUJrgVG+5lFINiR4RA+h
EMGXAshda+xwqm3KrdtHDLHRgFyfYov6w7s+caUMyV7gky1zmrB/NR+vG8Di2U2/
wyK+4y/c/Qt1CFhZSmuZ0zqzRzX7J2oxlT4P9FVdapnL5AYfXe6hZWhHJERjXmeS
GPovCQO/tBqRUiL9RwX6rcYbxykh9oseP4yxp5QZwLIO7cuIaStgMIN8z1vpZoBf
r7l3Bbd+ppRcq8NDqa7elRP0uiHm0wg7gMMcMWJJOJMU2Jm5DAw7PvZSA2FbksDL
Cu8WAloTsFCg11at6oTz6IcxdXsXTkpNp8O8Qv2yICj3Kw7gwX22Mc4V/CclrtFp
8lKFkTacJVbMkihgOFpu
=QN+b
-END PGP SIGNATURE-
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/15/2015 04:55 PM, James Page wrote:
Hi All
On 27/05/15 09:14, Thomas Goirand wrote:
tl;dr: - We'd like to push distribution packaging of OpenStack on
upstream gerrit with reviews. - The intention is to better share
the workload, and improve the overall QA for packaging *and*
upstream. - The goal is *not* to publish packages upstream -
There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get
started. - There's an ongoing discussion about using a distribution
specific namespace, my own opinion here is that using
/openstack-pkg-{deb,rpm} or /stackforge-pkg-{deb,rpm} would be the
most convenient because of a number of technical reasons like the
amount of Git repository. - Finally, let's not discuss for too long
and let's do it!!!
While working to re-align the dependency chain for OpenStack between
Debian and Ubuntu 15.10, and in preparation for the first Liberty
milestone, my team and I have been reflecting on the proposal to make
Ubuntu and Debian packaging of OpenStack a full OpenStack project.
We’ve come to the conclusion that for Debian and Ubuntu the best place
for us to collaborate on packaging is actually in the distributions,
as we do today, and not upstream in OpenStack.
Ubuntu has collaborated effectively with Debian since its inception
and has an effective set of tools to support the flow of packaging
(from Debian), bug reports and patches (to Debian) that have proven
effective, in terms of both efficiency and value, ever since I’ve been
working across both distributions.
This process allows each distribution to maintain its own direction,
whilst ensuring that any value that might be derived from
collaboration is supported with as minimal overhead as possible.
We understand and have communicated from the start of this
conversation that we will need to be able to maintain deltas between
Debian and Ubuntu - for both technical reasons, in the way the
distributions work (think Ubuntu main vs universe), as well as
objectives that each distribution has in terms of the way packaging
should work.
We don’t think that’s going to be made any easier by moving all of the
packaging under the OpenStack project - it just feels like we’re
trying to push a solved problem somewhere else, and then re-solve it
in a whole new set of ways.
The problem of managing delta and allowing a good level of
distribution independence is still going to continue to exist and will
be more difficult to manage due to the tighter coupling of development
process and teams than we have today.
On this basis, we're -1 on taking this proposal forward.
That said, we do appreciate that the Ubuntu packaging for OpenStack is
not as accessible as it might be using Bazaar as a VCS. In order to
provide a more familiar experience to developers and operators looking
to contribute to the wider Openstack ecosystem we will be moving our
OpenStack packaging branches over to the new Git support in Launchpad
in the next few weeks.
Regards
James
James,
During our discussions at the Summit, you seemed to be enthusiastic
about pushing our packaging to Stackforge. Then others told me to push
it to the /openstack namespace to make it more big tent-ish, which
made me very excited about the idea.
So far, I've been very happy of the reboot of our collaboration, and
felt like it was just awesome new atmosphere. So I have to admit I'm a
bit disappointed to read the above, even though I do understand the
reasoning.
Anyway, does this mean that you don't want to push packaging to
/stackforge either, which was the idea we shared at the summit?
I'm a bit lost on what I should do now, as what was exciting was
enabling operation people to contribute. I'll think about it and see
what to do next.
Cheers,
Thomas Goirand (zigo)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/15/2015 11:48 AM, Thomas Goirand wrote: On 06/15/2015 04:55 PM, James Page wrote: The problem of managing delta and allowing a good level of distribution independence is still going to continue to exist and will be more difficult to manage due to the tighter coupling of development process and teams than we have today. On this basis, we're -1 on taking this proposal forward. That said, we do appreciate that the Ubuntu packaging for OpenStack is not as accessible as it might be using Bazaar as a VCS. In order to provide a more familiar experience to developers and operators looking to contribute to the wider Openstack ecosystem we will be moving our OpenStack packaging branches over to the new Git support in Launchpad in the next few weeks. [...] During our discussions at the Summit, you seemed to be enthusiastic about pushing our packaging to Stackforge. Then others told me to push it to the /openstack namespace to make it more big tent-ish, which made me very excited about the idea. So far, I've been very happy of the reboot of our collaboration, and felt like it was just awesome new atmosphere. So I have to admit I'm a bit disappointed to read the above, even though I do understand the reasoning. James is right. This discussion thread put a lot of faith in the possibility that moving packaging efforts under the OpenStack umbrella would magically solve our key blocking issues. (I'm guilty of it as much as anyone else.) But really, we the collaborators are the ones who have to solve those blocking issues, and we'll have to do it together, no matter what banner we do it under. Anyway, does this mean that you don't want to push packaging to /stackforge either, which was the idea we shared at the summit? I'm a bit lost on what I should do now, as what was exciting was enabling operation people to contribute. I'll think about it and see what to do next. It doesn't really matter where the repos are located, we can still collaborate. Just moving Ubuntu's openstack repos to git and the Debian Python Modules Team repos to git will be a massive step forward. Allison __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/15/2015 03:03 PM, Allison Randal wrote: On 06/15/2015 11:48 AM, Thomas Goirand wrote: On 06/15/2015 04:55 PM, James Page wrote: The problem of managing delta and allowing a good level of distribution independence is still going to continue to exist and will be more difficult to manage due to the tighter coupling of development process and teams than we have today. On this basis, we're -1 on taking this proposal forward. That said, we do appreciate that the Ubuntu packaging for OpenStack is not as accessible as it might be using Bazaar as a VCS. In order to provide a more familiar experience to developers and operators looking to contribute to the wider Openstack ecosystem we will be moving our OpenStack packaging branches over to the new Git support in Launchpad in the next few weeks. [...] During our discussions at the Summit, you seemed to be enthusiastic about pushing our packaging to Stackforge. Then others told me to push it to the /openstack namespace to make it more big tent-ish, which made me very excited about the idea. So far, I've been very happy of the reboot of our collaboration, and felt like it was just awesome new atmosphere. So I have to admit I'm a bit disappointed to read the above, even though I do understand the reasoning. James is right. This discussion thread put a lot of faith in the possibility that moving packaging efforts under the OpenStack umbrella would magically solve our key blocking issues. (I'm guilty of it as much as anyone else.) But really, we the collaborators are the ones who have to solve those blocking issues, and we'll have to do it together, no matter what banner we do it under. Anyway, does this mean that you don't want to push packaging to /stackforge either, which was the idea we shared at the summit? I'm a bit lost on what I should do now, as what was exciting was enabling operation people to contribute. I'll think about it and see what to do next. It doesn't really matter where the repos are located, we can still collaborate. Just moving Ubuntu's openstack repos to git and the Debian Python Modules Team repos to git will be a massive step forward. While I agree those points are valid, and going to be helpful, moving under OpenStack (even Stackforge) does also offer the chance to get more test integration upstream (not saying this was the original scope). However, this could also be achieved by 3rd party integration too. I'm still driving forward with some -infra specific packaging for Debian / Fedora ATM (zuul packaging). Mostly because of -infra needs for packages. Not saying that is a reason to reconsider, but there is the need for -infra to consume packages from upstream. Thomas where does this leave you (Debian). Are you still considering the move to upstream? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/10/2015 04:31 PM, Joe Gordon wrote: On Wed, Jun 10, 2015 at 5:24 PM, Ian Cordasco ian.corda...@rackspace.com mailto:ian.corda...@rackspace.com wrote: On 6/10/15, 09:12, Thomas Goirand z...@debian.org mailto:z...@debian.org wrote: On 06/10/2015 12:25 PM, Dave Walker wrote: The initial core reviewers was seeded by representatives of distro's and vendors to get their input on viability in distro's. Really? James, were you made core on the requirements? I once tried to follow the requirements repo, though it moves too fast, and sometimes, it takes less than 2 days to get a new thing approved. Also, this repository has more than just dependencies, there's a lot of noise due to changes changes in projects.txt. I also don't mind any upgrade for whatever oslo or client libs. I'd love to have an easier way to voice my opinion without having all the noise of that repo in my inbox. I'm not sure if there's a solution to this problem though. Thomas You should be able to subscribe to a subset of the changes in gerrit. I don't recall if it only works for directories, but you should be able to make something work for *requirements.txt. The docs are easy to find on Google or DDG. Query to see only *requirements.txt changes: project:openstack/requirements file:^.*requirements.txt is:open how to subscribe to a subset of changes: https://review.openstack.org/Documentation/user-notify.html Cheers, Ian Ah, thanks so much!!! Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 10/06/15 12:07, Robert Collins wrote: On 10 June 2015 at 20:12, Matthias Runge mru...@redhat.com wrote: Since our software is to be consumed by packages, shouldn't the packages project consider itself to be responsible for global requirements? I.e. checking, if requirements are packageable, if versions fit, etc. I think we welcome input from distribution maintainers on global-requirements; especially for new packages. But, the responsibility is ultimately a team effort: all the components of openstack have to meet the operator/distributor co-installability requirement. If one project has a minimum version of X, its not possible for other projects to have a max version of X otherwise we're not coinstallable. This works both ways of course. My wording was not that good here. I know, some distro packagers are already looking at changes, but maybe this could be improved, i.e. intensified? It was more about: giving this more basic openstack effort a home in a project. In some distros, there are multiple versions of the same package allowed, in others, it's forbidden. Thats true, but its also a per-distro thing. Within a distro you need to be consistent. There's no need for RHEL to match RDO for instance, and trying to make that happen across a dozen redistributors in the OpenStack context makes no sense at all. We're moving to making our ranges as wide as we can to make life easier for anyone that wants to pick slightly different versions: we can't assert that it will work, but unless we know it doesnt', we won't preclude you trying :) Yes, that's right. But distros have an interest in being able to install the full stack somewhere. If we have conflicting requirements preventing that, it should be a target of the packaging project, to fix this. Currently, we (as OpenStack devs) are offloading those checks (and fixes) completely to distributions. Matthias __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 10 June 2015 at 20:12, Matthias Runge mru...@redhat.com wrote: Since our software is to be consumed by packages, shouldn't the packages project consider itself to be responsible for global requirements? I.e. checking, if requirements are packageable, if versions fit, etc. I think we welcome input from distribution maintainers on global-requirements; especially for new packages. But, the responsibility is ultimately a team effort: all the components of openstack have to meet the operator/distributor co-installability requirement. If one project has a minimum version of X, its not possible for other projects to have a max version of X otherwise we're not coinstallable. This works both ways of course. In some distros, there are multiple versions of the same package allowed, in others, it's forbidden. Thats true, but its also a per-distro thing. Within a distro you need to be consistent. There's no need for RHEL to match RDO for instance, and trying to make that happen across a dozen redistributors in the OpenStack context makes no sense at all. We're moving to making our ranges as wide as we can to make life easier for anyone that wants to pick slightly different versions: we can't assert that it will work, but unless we know it doesnt', we won't preclude you trying :) -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Hi Derek, I selected these 80 to move all of what RDO is currently maintaining on gerrithub to review.openstack.org, this was perhaps too big a set and in RDO we instead may need to go hybrid. Yeah, In my opinion we ahve lots of repeated divergence between the different python modules, so getting them sorted out in a small set of packages and then extending it to all python-* modules (and then as a 3rd step to all openstack-* modules) would be a better approach in my opinion (small steps). 1. pull what I've proposed (or a subset of it) into a rpm namespace and from there work in package to get them to a point where all rpm interested parties can use them. +1 3. Same as 2 but start with Suse packaging well, imho we should have a look at which starting point makes more sense for both of us. I see goods and bads in the spec files on both sides (of course my view is a bit biased on that :-) ). For this specific example I think differences of opinion are ok, we should provide the tools for each party interest in the packaging can hook in their own patches (I'm not sure what this would look like yet), I'm assuming here that we would also have deployer's out there interested who would have their own custom patches and bug fixes that they are interested in. Right, that might be a good solution (use pristine upstream packaging and provide tools for downstream to modify/add patches easily). For most of the python-* dependencies we have zero patches so its not a big issue for the initial step, it gets more complex as we get closer to the python*clients and openstack* packages, as we tend to have a need for patches in there that have not yet been merged upstream (for whatever reason). +1, maybe we should schedule something in a few days where we could go though the differences of a specific package and how things could take shape. Good idea. Whats a good time and date? I'm mostly available tomorrow-Sunday in the CEST time zone. Greetings, Dirk __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 10 June 2015 at 11:07, Robert Collins robe...@robertcollins.net wrote: On 10 June 2015 at 20:12, Matthias Runge mru...@redhat.com wrote: Since our software is to be consumed by packages, shouldn't the packages project consider itself to be responsible for global requirements? I.e. checking, if requirements are packageable, if versions fit, etc. I think we welcome input from distribution maintainers on global-requirements; especially for new packages. But, the responsibility is ultimately a team effort: all the components of openstack have to meet the operator/distributor co-installability requirement. If one project has a minimum version of X, its not possible for other projects to have a max version of X otherwise we're not coinstallable. This works both ways of course. In some distros, there are multiple versions of the same package allowed, in others, it's forbidden. Thats true, but its also a per-distro thing. Within a distro you need to be consistent. There's no need for RHEL to match RDO for instance, and trying to make that happen across a dozen redistributors in the OpenStack context makes no sense at all. We're moving to making our ranges as wide as we can to make life easier for anyone that wants to pick slightly different versions: we can't assert that it will work, but unless we know it doesnt', we won't preclude you trying :) -Rob Just to add some history here, this was *precisely* the problems that vendors were having - but worse, each openstack project had conflicting version requirements making it really quite hard for distro's to centralise package this.. This is why the project, openstack/requirements was created to centralise the management of this to avoid conflicting version requirements AND get input back from distro's and vendors. The initial core reviewers was seeded by representatives of distro's and vendors to get their input on viability in distro's. Vendors and distro's didn't engage as much as they could have (myself included), which means that they had less input. It is pretty easy to get that input back, you just need to review the incoming changes: https://review.openstack.org/#/q/status:open+project:openstack/requirements+branch:master,n,z I did start working on a jenkins job to check distro's to see what version of package was already in releases, but it wasn't really reliable enough, so I dropped it on the floor. If you wanted to work on a jenkins job to provide advise on proposed changesets, I am sure the infra' team would be supportive. -- Kind Regards, Dave Walker __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Excerpts from Matthias Runge's message of 2015-06-10 12:29:45 +0200: On 10/06/15 12:07, Robert Collins wrote: On 10 June 2015 at 20:12, Matthias Runge mru...@redhat.com wrote: Since our software is to be consumed by packages, shouldn't the packages project consider itself to be responsible for global requirements? I.e. checking, if requirements are packageable, if versions fit, etc. I think we welcome input from distribution maintainers on global-requirements; especially for new packages. But, the responsibility is ultimately a team effort: all the components of openstack have to meet the operator/distributor co-installability requirement. If one project has a minimum version of X, its not possible for other projects to have a max version of X otherwise we're not coinstallable. This works both ways of course. My wording was not that good here. I know, some distro packagers are already looking at changes, but maybe this could be improved, i.e. intensified? It was more about: giving this more basic openstack effort a home in a project. Given the history of lack of interest, I would want to see a significant increase in reviews from that team before giving them control of the repository. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 27/05/15 10:14, Thomas Goirand wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi all,
tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the overall
QA for packaging *and* upstream.
- - The goal is *not* to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)
Since our software is to be consumed by packages, shouldn't the packages
project consider itself to be responsible for global requirements? I.e.
checking, if requirements are packageable, if versions fit, etc.
In some distros, there are multiple versions of the same package
allowed, in others, it's forbidden.
Matthias
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/10/2015 12:25 PM, Dave Walker wrote: The initial core reviewers was seeded by representatives of distro's and vendors to get their input on viability in distro's. Really? James, were you made core on the requirements? I once tried to follow the requirements repo, though it moves too fast, and sometimes, it takes less than 2 days to get a new thing approved. Also, this repository has more than just dependencies, there's a lot of noise due to changes changes in projects.txt. I also don't mind any upgrade for whatever oslo or client libs. I'd love to have an easier way to voice my opinion without having all the noise of that repo in my inbox. I'm not sure if there's a solution to this problem though. Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 10/06/15 15:12, Thomas Goirand wrote: The initial core reviewers was seeded by representatives of distro's and vendors to get their input on viability in distro's. Really? James, were you made core on the requirements? Believe it or not, I've not *always* worked on OpenStack for Ubuntu :-); that pre-dates my direct involvement. - -- James Page Ubuntu and Debian Developer james.p...@ubuntu.com jamesp...@debian.org -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBCAAGBQJVeEgKAAoJEL/srsug59jDrqEQAK6jXUCDzHb8XIVM6GM82QOy +6NXp99us8xDzq/2mWuZAUZ4abon4JvZmOjngSoQmTXs1e9L3HiQr4iTlW4ZlJ9w JCL8Xlq2/SJL8KJ2hMFg6sWanJZ0eglJ21F/bq2Rz6/fjhx/i7q8CPLHJghG5RXm 2sk9seXHFLK3syW4g/sVmz3MkPAYwu6ZqFiTDDcgNjkD+OegoMltn/d+HRLf20G9 VeTuL6mjW2ZnnJozBuqJ+ZbX+Gc35OhV8NWFzhaPkMPNT48BULr7rpfAgQ22WXyP v9EnXaYtzyhUQBOdyrKXoPLK2dhCTcqRGImM1lZl3mKJbG/jf8d9U24M20bw4kQE h6LHOyH+PQQcsAm+uKH2mpB8c+XElmeL7BO9sSUsC30oL0QI3OxRcFt/wgIycHwd f8OoXPc8RsZERlsWYPF9gA16bTRQn08vIgVweNKu7vkP5qR6nvOjCzRBoUZdlIpu 4tqlzprYDSzSAJi8mzDq+nw3YulXjuTk/vQusp4MCjfA3VXS1yry8i+1HpjF/MO/ eAoKpZM2ntksRcObj947yh7ncsxjrQmOXHPfqMlbZTa6NWmQnElufJJ0nk5s30el cBxUTjLN4lvAc6xjZPXqHk1U0kiP95E0wAa3EMVSDLeULG4Hq57+bSE63zHTts77 Y+UyyJswgeA6ZHC+W5Ct =egET -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 10 June 2015 at 15:12, Thomas Goirand z...@debian.org wrote: On 06/10/2015 12:25 PM, Dave Walker wrote: The initial core reviewers was seeded by representatives of distro's and vendors to get their input on viability in distro's. Really? James, were you made core on the requirements? I once tried to follow the requirements repo, though it moves too fast, and sometimes, it takes less than 2 days to get a new thing approved. Also, this repository has more than just dependencies, there's a lot of noise due to changes changes in projects.txt. I also don't mind any upgrade for whatever oslo or client libs. I'd love to have an easier way to voice my opinion without having all the noise of that repo in my inbox. I'm not sure if there's a solution to this problem though. On the Ubuntu side, I believe Chuck and myself were carrying the flag (we had +2). When the openstack/requirements project was created James was less involved upstream, and two representatives of a distro ought to have been enough. An yes, I also found that the flow was too much to keep up with which is why I tried to automate some of this. I hoped the burden has reduced somewhat, as prospective changes now need to do more of the distro research themselves. Is the library already packaged in the distros we target (Ubuntu latest / Fedora latest)? By adding something to OpenStack global-requirements.txt we are basically demanding that Linux Distros package this for the next release of OpenStack. If they already have, great. If not, we should be cautious of adding it.:ref:`finding-distro-status`[0] But only so much can be done by non-distro focussed upstream consumers... [0] https://github.com/openstack/requirements/blob/master/README.rst#for-new-requirements -- Kind Regards, Dave Walker __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 6/10/15, 09:12, Thomas Goirand z...@debian.org wrote: On 06/10/2015 12:25 PM, Dave Walker wrote: The initial core reviewers was seeded by representatives of distro's and vendors to get their input on viability in distro's. Really? James, were you made core on the requirements? I once tried to follow the requirements repo, though it moves too fast, and sometimes, it takes less than 2 days to get a new thing approved. Also, this repository has more than just dependencies, there's a lot of noise due to changes changes in projects.txt. I also don't mind any upgrade for whatever oslo or client libs. I'd love to have an easier way to voice my opinion without having all the noise of that repo in my inbox. I'm not sure if there's a solution to this problem though. Thomas You should be able to subscribe to a subset of the changes in gerrit. I don't recall if it only works for directories, but you should be able to make something work for *requirements.txt. The docs are easy to find on Google or DDG. Cheers, Ian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On Wed, Jun 10, 2015 at 5:24 PM, Ian Cordasco ian.corda...@rackspace.com wrote: On 6/10/15, 09:12, Thomas Goirand z...@debian.org wrote: On 06/10/2015 12:25 PM, Dave Walker wrote: The initial core reviewers was seeded by representatives of distro's and vendors to get their input on viability in distro's. Really? James, were you made core on the requirements? I once tried to follow the requirements repo, though it moves too fast, and sometimes, it takes less than 2 days to get a new thing approved. Also, this repository has more than just dependencies, there's a lot of noise due to changes changes in projects.txt. I also don't mind any upgrade for whatever oslo or client libs. I'd love to have an easier way to voice my opinion without having all the noise of that repo in my inbox. I'm not sure if there's a solution to this problem though. Thomas You should be able to subscribe to a subset of the changes in gerrit. I don't recall if it only works for directories, but you should be able to make something work for *requirements.txt. The docs are easy to find on Google or DDG. Query to see only *requirements.txt changes: project:openstack/requirements file:^.*requirements.txt is:open how to subscribe to a subset of changes: https://review.openstack.org/Documentation/user-notify.html Cheers, Ian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/09/2015 10:20 AM, James Page wrote: LGTM - although for any initial repository migration, I'd like to see Ubuntu (from bzr) and Debian (git.debian.org) branches separately for projects that have Vcs branches for Ubuntu so that we can manage that delta I keep going on about effectively; that would include: ceilometer cinder designate glance heat horizon ironic keystone neutron neutron-fwaas neutron-lbaas neutron-vpnaas nova openstack-trove sahara swift Apart from the VCS and Maintainer: field, what difference do we have in the below packages? neutron-fwaas neutron-lbaas neutron-vpnaas swift Cheers, Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Hi Derek, 2015-06-09 0:34 GMT+02:00 Derek Higgins der...@redhat.com: This patch would result in 80 packaging repositories being pulled into gerrit. I personally would prefer to start with fewer but common packages between all RPM distros (is there more than Red Hat and SUSE ?) than starting the process with 80, but I wouldn't object to that. o exactly what namespace/prefix to use in the naming, I've seen lots of opinions but I'm not clear if we have come to a decision o Should we use rdo in the packaging repo names and not rpm? I think this ultimatly depends whether the packaging can be shared between RDO and Suse or not. Well, we're (SUSE that is) are interested in sharing the packaging, and a non-RDO prefix would be preferred for the upstream coordination efforts. It is all a bit fuzzy for me right now as I'm not entirely sure our goals for packaging are necessarily the same (e.g. we have the tendency to include patches that have not been merged but are proposed upstream and are +1'ed already into our packages should there be a pressing need for us to do so (e.g. fixes an important platform bug), but maybe we can find enough common goals to make this a benificial effort for all of us. There are quite some details to sort out as our packaging is for historical and for various policy reasons that we need to stick to slightly different than the RDO packaging. I think going over those and see how we can merge them in a consolidated effort (or maintain two variants together) is the first step IMHO. Another important point for us is that we start with equal rights on the upstream collaboration (at least on the RPM side, I am fine with decoupling and not caring about the deb parts). I'm not overly optimistic that a single PTL would be able to cover both the DEB and RPM worlds, as I perceive them quite different in details. Greetings, Dirk __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 08/06/15 22:36, Thomas Goirand wrote: [...] I have sorted this list into categories, and sorted these categories in an increasing order of likelihood to be maintained in upstream gerrit. On the below list, I believe we should have in upstream gerrit, at least: - OpenStack maintained libraries and clients - Debian specific packages (because that's needed tools for building and running a Debian package based CI) - server packages All the 3rd party Python modules could either stay within the PKG OpenStack Debian team, or move to the DPMT (Debian Python Module Team). Though I will *refuse* that these packages are switched from Git to SVN, so we will have to wait until the DPMT finishes the switch. I've heard that Tumbleweed (that's a nick name...) is close to have this migration finished though. I understand the migration to be nearing completion as well, which unblocks any team repository migration activity. Also, probably it would make sense to keep some of the tooling within the PKG OpenStack group. I'm thinking about all the unit test stuff, like testr, subunit, and all of its dependencies (testtools, testscenarios, etc.). Maybe it's a good fit for upstream packaging too? Please voice your opinion here. I'd be tempted to leave then in Debian under the DPMT - they don't release on the same cadence as OpenStack afaik - so it may be easier to just collaborate in-distro on that stuff. Suggestion - leave them out of the migration for now - we can always include them later if the requirement/need arises. I would then like to keep side packages and Key dependencies within the PKG OpenStack group in alioth.debian.org. side packages - +1 fine with me key dependencies - see below. This overall means that we'd push 107 repositories to Gerrit, and even 119 if we include TripleO. And of course, this list would grow over time (because that's OpenStack you know... things always grow, and never shrink...). It took me some time to produce this list below. I hope that's useful. It is - thank-you for doing this work. Key dependencies (4): - alembic migrate I'd put these two as candidates for migration to the DPMT. novnc rabbitmq-server OK - these two are fine where they are. 3rd party Python modules (101): --- [...] testresources websockify LGTM TripleO (12): - python-dib-utils python-diskimage-builder python-os-apply-config python-os-client-config python-os-cloud-config python-os-collect-config python-os-net-config python-os-refresh-config tripleo-heat-templates tripleo-image-elements tuskar tuskar-ui server packages (25): - barbican ceilometer cinder designate glance heat heat-cfntools horizon ironic keystone murano murano-agent murano-dashboard networking-arista networking-mlnx neutron neutron-fwaas neutron-lbaas neutron-vpnaas nova openstack-trove rally sahara swift swift-plugin-s3 LGTM - although for any initial repository migration, I'd like to see Ubuntu (from bzr) and Debian (git.debian.org) branches separately for projects that have Vcs branches for Ubuntu so that we can manage that delta I keep going on about effectively; that would include: ceilometer cinder designate glance heat horizon ironic keystone neutron neutron-fwaas neutron-lbaas neutron-vpnaas nova openstack-trove sahara swift Debian specific packages (3): - openstack-debian-images openstack-meta-packages openstack-pkg-tools OpenStack maintained libraries and clients (79): openstack-doc-tools openstack-nose oslo-config oslo-sphinx oslo.messaging oslo.rootwrap python-barbicanclient python-bashate python-ceilometerclient python-cinderclient python-congressclient python-debtcollector python-designateclient python-django-openstack-auth python-glance-store python-glanceclient python-hacking python-heatclient python-hp3parclient python-hplefthandclient python-ironicclient python-keystoneclient python-keystonemiddleware python-mistralclient python-muranoclient python-neutronclient python-novaclient python-openstackclient python-oslo-context python-oslo.concurrency python-oslo.db python-oslo.i18n python-oslo.log python-oslo.middleware python-oslo.policy python-oslo.serialization python-oslo.utils python-oslo.versionedobjects python-oslo.vmware python-oslotest python-osprofiler python-pbr python-proliantutils python-pycadf python-pyeclib python-saharaclient python-savannaclient python-swiftclient python-taskflow python-tempest-lib python-tooz python-troveclient python-tuskarclient python-xstatic python-xstatic-angular python-xstatic-angular-bootstrap python-xstatic-angular-cookies python-xstatic-angular-lrdragndrop python-xstatic-angular-mock python-xstatic-bootstrap-datepicker python-xstatic-bootstrap-scss python-xstatic-d3
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/09/2015 05:37 AM, Dirk Müller wrote: Hi Derek, 2015-06-09 0:34 GMT+02:00 Derek Higgins der...@redhat.com: This patch would result in 80 packaging repositories being pulled into gerrit. I personally would prefer to start with fewer but common packages between all RPM distros (is there more than Red Hat and SUSE ?) than starting the process with 80, but I wouldn't object to that. I agree, I would start with a limit set for the first pass. Especially since people haven't decided on the naming schema yet. o exactly what namespace/prefix to use in the naming, I've seen lots of opinions but I'm not clear if we have come to a decision o Should we use rdo in the packaging repo names and not rpm? I think this ultimatly depends whether the packaging can be shared between RDO and Suse or not. Well, we're (SUSE that is) are interested in sharing the packaging, and a non-RDO prefix would be preferred for the upstream coordination efforts. It is all a bit fuzzy for me right now as I'm not entirely sure our goals for packaging are necessarily the same (e.g. we have the tendency to include patches that have not been merged but are proposed upstream and are +1'ed already into our packages should there be a pressing need for us to do so (e.g. fixes an important platform bug), but maybe we can find enough common goals to make this a benificial effort for all of us. I agree too, rdo prefix is too specific in this case for a repo name. There are quite some details to sort out as our packaging is for historical and for various policy reasons that we need to stick to slightly different than the RDO packaging. I think going over those and see how we can merge them in a consolidated effort (or maintain two variants together) is the first step IMHO. Another important point for us is that we start with equal rights on the upstream collaboration (at least on the RPM side, I am fine with decoupling and not caring about the deb parts). I'm not overly optimistic that a single PTL would be able to cover both the DEB and RPM worlds, as I perceive them quite different in details. Greetings, Dirk __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 09/06/15 10:37, Dirk Müller wrote: Hi Derek, 2015-06-09 0:34 GMT+02:00 Derek Higgins der...@redhat.com: This patch would result in 80 packaging repositories being pulled into gerrit. I personally would prefer to start with fewer but common packages between all RPM distros (is there more than Red Hat and SUSE ?) than starting the process with 80, but I wouldn't object to that. I selected these 80 to move all of what RDO is currently maintaining on gerrithub to review.openstack.org, this was perhaps too big a set and in RDO we instead may need to go hybrid. o exactly what namespace/prefix to use in the naming, I've seen lots of opinions but I'm not clear if we have come to a decision o Should we use rdo in the packaging repo names and not rpm? I think this ultimatly depends whether the packaging can be shared between RDO and Suse or not. Well, we're (SUSE that is) are interested in sharing the packaging, and a non-RDO prefix would be preferred for the upstream coordination efforts. +1, I'd also like to see us share packaging so a non-RDO prefix should be avoided. I think we have a few possibilities here 1. pull what I've proposed (or a subset of it) into a rpm namespace and from there work in package to get them to a point where all rpm interested parties can use them. 2. pull them into an rdo namespace and from there work on convergence, as each package becomes usable by all interested parties we rename to rpm- I know renaming is a PITA for infra so maybe move to Attic and import a new repo if its easier. 3. Same as 2 but start with Suse packaging It is all a bit fuzzy for me right now as I'm not entirely sure our goals for packaging are necessarily the same (e.g. we have the tendency to include patches that have not been merged but are proposed upstream and are +1'ed already into our packages should there be a pressing need for us to do so (e.g. fixes an important platform bug), but maybe we can find enough common goals to make this a benificial effort for all of us. For this specific example I think differences of opinion are ok, we should provide the tools for each party interest in the packaging can hook in their own patches (I'm not sure what this would look like yet), I'm assuming here that we would also have deployer's out there interested who would have their own custom patches and bug fixes that they are interested in. But yes, there will be other differences that I'm sure we'll have to figure out. There are quite some details to sort out as our packaging is for historical and for various policy reasons that we need to stick to slightly different than the RDO packaging. I think going over those and see how we can merge them in a consolidated effort (or maintain two variants together) is the first step IMHO. +1, maybe we should schedule something in a few days where we could go though the differences of a specific package and how things could take shape. Another important point for us is that we start with equal rights on the upstream collaboration (at least on the RPM side, I am fine with decoupling and not caring about the deb parts). I'm not overly optimistic that a single PTL would be able to cover both the DEB and RPM worlds, as I perceive them quite different in details. yup, seems reasonable to me Greetings, Dirk __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/08/2015 10:39 AM, James Page wrote: On 02/06/15 23:41, James E. Blair wrote: 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. Having taken some time to think about this over the weekend, I'm keen to ensure that any packaging repositories that move upstream are packaging for OpenStack and other OpenStack umbrella projects. Thomas - how many of the repositories under the pkg-openstack team in Debian fall into this category - specifically projects under /openstack or /stackforge namespaces? I don't think we should be upstreaming packaging for the wider OpenStack dependency chain - the Debian Python modules team is a much larger team of interested contributors and better place for this sort of work. Ok, let's work this list of packages out. The full list of packages currently maintained within the Debian OpenStack PKG team is here: https://qa.debian.org/developer.php?login=openstack-devel%40lists.alioth.debian.org I have sorted this list into categories, and sorted these categories in an increasing order of likelihood to be maintained in upstream gerrit. On the below list, I believe we should have in upstream gerrit, at least: - OpenStack maintained libraries and clients - Debian specific packages (because that's needed tools for building and running a Debian package based CI) - server packages All the 3rd party Python modules could either stay within the PKG OpenStack Debian team, or move to the DPMT (Debian Python Module Team). Though I will *refuse* that these packages are switched from Git to SVN, so we will have to wait until the DPMT finishes the switch. I've heard that Tumbleweed (that's a nick name...) is close to have this migration finished though. Also, probably it would make sense to keep some of the tooling within the PKG OpenStack group. I'm thinking about all the unit test stuff, like testr, subunit, and all of its dependencies (testtools, testscenarios, etc.). Maybe it's a good fit for upstream packaging too? Please voice your opinion here. I would then like to keep side packages and Key dependencies within the PKG OpenStack group in alioth.debian.org. This overall means that we'd push 107 repositories to Gerrit, and even 119 if we include TripleO. And of course, this list would grow over time (because that's OpenStack you know... things always grow, and never shrink...). It took me some time to produce this list below. I hope that's useful. Cheers, Thomas Goirand (zigo) side packages (7): cobbler ftp-cloudfs git-review ntpstat q-text-as-data sftpcloudfs sheepdog Key dependencies (4): - alembic migrate novnc rabbitmq-server 3rd party Python modules (101): --- cliff-tablib factory-boy python-aioeventlet python-autobahn python-cloudfiles python-coffin python-colander python-concurrent.futures python-couleur python-crcmod python-croniter python-daemonize python-ddt python-django-appconf python-django-bootstrap-form python-django-compressor python-django-discover-runner python-django-pyscss python-dogpile.cache python-dogpile.core python-eventlet python-extras python-falcon python-gabbi python-greenio python-happybase python-httpretty python-hurry.filesize python-ibm-db-sa python-invocations python-invoke python-jingo python-json-patch python-json-pointer python-jsonpath-rw python-jsonrpclib python-jsonschema python-kafka python-ldappool python-lesscpy python-logutils python-misaka python-mockito python-mox3 python-nose-exclude python-nose-parameterized python-nose-testconfig python-nose-timer python-nosehtmloutput python-pecan python-pint python-posix-ipc python-proboscis python-protorpc-standalone python-pyghmi python-pygit2 python-pykmip python-pymemcache python-pymysql python-pysaml2 python-pyvmomi python-rednose python-requestbuilder python-requests-kerberos python-requests-mock python-retrying python-rfc3986 python-rtslib-fb python-rudolf python-scciclient python-seamicroclient python-semantic-version python-semver python-sockjs-tornado python-sphinxcontrib.plantuml python-steadymark python-sure python-sysv-ipc python-tablib python-tasklib python-termcolor python-termstyle python-testscenarios python-trollius python-txaio python-warlock python-wrapt python-wsgi-intercept python-wsme python-xmlbuilder python-xvfbwrapper python-yaql python-zake sphinxcontrib-docbookrestapi sphinxcontrib-httpdomain sphinxcontrib-pecanwsme sphinxcontrib-programoutput spice-html5 subunit testresources websockify TripleO (12): - python-dib-utils python-diskimage-builder python-os-apply-config python-os-client-config python-os-cloud-config python-os-collect-config python-os-net-config python-os-refresh-config tripleo-heat-templates tripleo-image-elements tuskar tuskar-ui server packages (25): - barbican ceilometer cinder designate glance
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 03/06/15 17:28, Haïkel wrote: 2015-06-03 17:23 GMT+02:00 Thomas Goirand z...@debian.org: i On 06/03/2015 12:41 AM, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. I've just read the IRC logs. And there's one thing I would like to make super clear. I still haven't read the logs as we had our post-mortem meeting today, but I'll try to address your points. We, ie: Debian Ubuntu folks, are very much clear on what we want to achieve. The project has been maturing in our heads for like more than 2 years. We would like that ultimately, only a single set of packages Git repositories exist. We already worked on *some* convergence during the last years, but now we want a *full* alignment. We're not 100% sure how the implementation details will look like for the core packages (like about using the Debconf interface for configuring packages), but it will eventually happen. For all the rest (ie: Python module packaging), which represent the biggest work, we're already converging and this has zero controversy. Now, the Fedora/RDO/Suse people jumped on the idea to push packaging on the upstream infra. Great. That's socially tempting. But technically, I don't really see the point, apart from some of the infra tooling (super cool if what Paul Belanger does works for both Deb+RPM). Finally, indeed, this is not totally baked. But let's please not delay the Debian+Ubuntu upstream Gerrit collaboration part because of it. We would like to get started, and for the moment, nobody is approving the /stackforge/deb-openstack-pkg-tools [1] new repository because we're waiting on the TC decision. First, we all agree that we should move packaging recipes (to use a neutral term) and reviewing to upstream gerrit. That should *NOT* be delayed. We (RDO) are even willing to transfer full control of the openstack-packages namespace on github. If you want to use another namespace, it's also fine with us. Then, about the infra/tooling things, it looks like a misunderstanding. If we don't find an agreement on these topics, it's perfectly fine and should not prevent moving to upstream gerrit So let's break the discussion in two parts. 1. upstream gerrit shared by everyone and get this started asap In an attempt to document how this would look for RDO, I've started a patch[1] that I'll iterate on while this discussions converges on a solution that will work. This patch would result in 80 packaging repositories being pulled into gerrit. I've left a TODO in the commit message to track questions I believe we still have to answer, most notably o exactly what namespace/prefix to use in the naming, I've seen lots of opinions but I'm not clear if we have come to a decision o Should we use rdo in the packaging repo names and not rpm? I think this ultimatly depends whether the packaging can be shared between RDO and Suse or not. o Do the RDO packaging repo's fall under this project[2] or is it its own group [1] https://review.openstack.org/#/c/189497 [2] https://review.openstack.org/#/c/185187 2. continue discussion about infra/tooling within the new project, without presumin the outcome. Does it look like a good compromise to you? Regards, H. Cheers, Thomas Goirand (zigo) [1] https://review.openstack.org/#/c/185164/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thomas On 03/06/15 16:23, Thomas Goirand wrote: This came up at the TC meeting today, and I volunteered to provide an update from the discussion. I've just read the IRC logs. And there's one thing I would like to make super clear. We, ie: Debian Ubuntu folks, are very much clear on what we want to achieve. The project has been maturing in our heads for like more than 2 years. We would like that ultimately, only a single set of packages Git repositories exist. We already worked on *some* convergence during the last years, but now we want a *full* alignment. Actually I think we agreed to work towards alignment on the dependency chain for OpenStack at the summit; alignment on the core packages (bits of OpenStack you actually install) is going to be much more challenging and might be something that we can't get full alignment on between Ubuntu and Debian - so let's not jump the gun on having that as a core objective for the packaging team. As I've stated before, we will have to have some way of managing that delta from the outset of moving any core packaging under the OpenStack umbrella. The Ubuntu packaging is used widely by end-users and a number of other projects including the OpenStack Puppet and Chef modules as well as the Juju charms for OpenStack - any changes to structure and behaviour are going to have much wider impact and I'm keen to ensure that we consider the requirements of our end users before we consider any full convergence objectives. - -- James Page Ubuntu and Debian Developer james.p...@ubuntu.com jamesp...@debian.org -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBCAAGBQJVdVGyAAoJEL/srsug59jDs8YP/2dJJHaXpn++80fOGg3S2z3Q 0ifzNvgMQNPDGGLnfZ95Q/8iWXhqF03waNcd33MLI0+HKFtuBaTZ7P/W3ImDDpEE B7TdwNGLzFN76Gz8Q9q0K+/6SxYGuwiWwlHrzJLaK4mEer83oojQ2v3Jxgw2SNx2 3UWamoFJm5o1s3Nh84QkpiXOQLZ51J1YjWXS0zz7gfqtgeiQvCr67l6dqJ/RaKGv B1oWeV4nT+yAogWx/7VX5Vzywab5Vo1PmIRLAC6BX9mKEeqoFOAZC7bd+DUNNp/J Rzg3KKQbSvXhL+xO0eByuWt4JE3EBJrI2bUz3LzutvWET5eJWnMY2gm1RmPcjguu LFjNKF/Bmjgzk88kTF3k8kBgghhR0FKJyFfYi14j8RshgIh6ghtzfnHcyamsiaQe 8fitDq/k5p0u6F9zplJ2U4wYptV0COkwlcJTSOrvACQnziCOBM+k6VE2bcLqS5wC kFnw/0I0iIycKFYxqvSBhR+fnWHQIXD9Swvh6EhF/VT6TQRKxIY2pOci/JYo+/Zv rLTAfoKxmtlw8HOjifcLQSem7YoxF2O9qgUNqVkzg6g6PzpXAK4S8LnEq3on9v7p wRJvFhmzfuo3xvtlsdvbTdea0VzRXCG0SM+XfEzB4FSvk4jhWI4TOuqiMxStq4SQ QyZ3zzq0M0f5uEQ/xQZV =zgNr -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/04/2015 01:25 AM, Thomas Goirand wrote:
On 06/03/2015 08:07 PM, Andreas Jaeger wrote:
On 06/03/2015 03:57 PM, James Page wrote:
[...]
After some discussion with Thomas on IRC, I think this is more than
one effort; The skills and motivation for developers reviewing
proposed packaging changes needs to be aligned IMO - so I think it
makes sense to split the packaging teams between:
Debian/Ubuntu + derivatives
CentOS/Fedora/RHEL + derivatives
So, would this be two packaging teams - on DEB team and one RPM team?
Yes.
How would those two teams collaborate - or is no collaboration needed?
I don't see what there would be collaboration on. Even the thing we work
on is called differently (ie: specs in Red Hat, and source packages
in Debian/Ubuntu). Though if there are things on which we can converge
(like package names maybe?), it would be a good thing for final users.
The control files (specs, deb) are indeed different, the question is
what they can share.
I see collaboration possibilities on package names and layout - like how
to split a package up -, configuration files, defaults... This doesn't
need to be a first step but something to consider when packaging new
repositories.
How are you handling Debian and Ubuntu differences?
There's not so much difference already on the Python module
dependencies, as Ubuntu imports packages from Debian. They imported 100%
of my work for Juno, then diverged for Kilo since I didn't upload (as
Jessie was frozen). Now that's the only difference we need to fix, and
James is already working on that.
We already work on some packages together (like James Page and myself
are co-maintaining rabbitmq-server (he did most of it, shame on me)).
As for the core OpenStack packages, what I wrote works already on Ubuntu
if you rebuild the package, though it may need a few fix here and there
for version dependencies (as Jessie was frozen after Trusty), but that's
easy to fix. The harder part will be for Neutron and Nova. In case we
don't agree at all, we can still use dpkg-vendors --derive-from ubuntu
to check on which distribution we are, and do actions from there.
There's already such mechanism in the debian/rules file of the Debian
Nova package for example (to check on the group name for libvirt which
differs in Debian Ubuntu). It's not in the Ubuntu version of the
package, but I'm sure that it will soon land there! :)
The only thing we can't fix will be the Maintainer: and Uploaders:
field, but even that, maybe we can find a way so we don't have to fix
it either.
So in a smaller amount of words: I'm sure we can find a way to reduce
our differences to the absolute zero on the source package level, even
though the result may look different thanks to some conditionals where
it can't be avoided.
Understood,
Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton,
HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 02/06/15 23:41, James E. Blair wrote: 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. Having taken some time to think about this over the weekend, I'm keen to ensure that any packaging repositories that move upstream are packaging for OpenStack and other OpenStack umbrella projects. Thomas - how many of the repositories under the pkg-openstack team in Debian fall into this category - specifically projects under /openstack or /stackforge namespaces? I don't think we should be upstreaming packaging for the wider OpenStack dependency chain - the Debian Python modules team is a much larger team of interested contributors and better place for this sort of work. - -- James Page Technical Architect OpenStack Engineering Team james.p...@canonical.com -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBCAAGBQJVdVSwAAoJEL/srsug59jDGQ8P/jiupTb6Sx48QHAOkPSlJJBj IFfLY/JLpV+lsdizaIEPH1S232qWg7JtPTVeRDYfEXGtRN+5whJD505frxmTesEN W68LVMSybbIPIf8f++MMm6PZ1d0wq9STHf8Mi0PvzkjUcDO1xDASNzG2ZRh7X56C bavfGXAxiYkIdzqB5lUTXoYhcKuMTFq0YzIIQv6Ebgst1hqeOCsEcXYvIEnOTVyM Dgfc94h13+1+cSSwpo5ilcbwD2uAgsEDQBz2hFPShG7CigW6dY/454hXPCtDyWf7 alLgawiWYsQUmfjCz1ozVvzqrMoCFF1rFqlzlfXUTZc9obKDHtWzorO5KwTkkmRw AIvnj/zYAlFBPDTDG84ZFVDAH9EJ2y5x3NiIoe16lLP67PGQiUO7BYJCoSQeZT2N 9axsihKmgdwm2icYBrmIFKS83aKUjAoYZ/TZd4yKYVqbdOtG2ysDAt7REXU63UZN 12KArxuFR8ygJS1Kl0MkRDOMX9omWJ8R5EgcEhjtbW6s0XBaaK38vCZECtCmnaBh nEl+XHV/4GUxpNa424dvwWm8VAm+e0nIFycCBXqgpqdlpDGmAbLdX/B+Kygw5IEY oI8WsEWbbHKBqdAuYU4ZUxHbUcYxzZ/mYwT9L5M/2ETOrOGsQRRJlxigUbgj4BVY R/FemtQlWHCdbj3uDac3 =nMQI -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 04/06/15 22:54, Thomas Goirand wrote: The init scripts used to be hard to maintain because they were many, but since Debian Ubuntu are using automatic generation out of a tiny template (with sysv-rc, systemd and upstart all supported), this is a problem solved. Ooh, that sounds like something that my team would like to use. Please could you point to more information about this tool? Thanks, Neil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/08/2015 10:26 AM, James Page wrote: The Ubuntu packaging is used widely by end-users and a number of other projects including the OpenStack Puppet and Chef modules as well as the Juju charms for OpenStack - any changes to structure and behaviour are going to have much wider impact and I'm keen to ensure that we consider the requirements of our end users before we consider any full convergence objectives. I don't know for the Chef and Juju charms projects, but for puppet, I know that they do work with the Debian packages (also because I added a bunch of Provides:). When I was working with Emilien Macchi, he made it so it would. For a number of packages, it will be trivial. Only for ultra-complex cases, like Nova and Neutron, it's going to be harder. But I'm sure we can achieve something for packages like Cinder, Glance, etc. Now, I started this proposal so that we could work more together. I hope you aren't backing-up on the promise, and that such an effort to align all will happen. Sure, announce less, deliver more, but at the same time, we need a declaration of intend. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/08/2015 08:32 AM, Andreas Jaeger wrote: The control files (specs, deb) are indeed different, the question is what they can share. I see collaboration possibilities on package names and layout - like how to split a package up -, configuration files, defaults... This doesn't need to be a first step but something to consider when packaging new repositories. Agreed. However, RPM packaging came later, and decided to prefix each server project with openstack-. Also, RPM packages understand upper/lower case, when the deb format explicitly forbids it. So we'd have to have some kind of change in the policy for RPM packages to happen first, before we have the same package names. I don't see how this can happen. As for splitting-up a package into multiple ones, this is more or less dictated by upstream (for example: sahara-api sahara-engine are the 2 daemons so we do 2 packages). So all together, I don't see much we could do that we don't already. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/08/2015 05:29 PM, Neil Jerram wrote: On 04/06/15 22:54, Thomas Goirand wrote: The init scripts used to be hard to maintain because they were many, but since Debian Ubuntu are using automatic generation out of a tiny template (with sysv-rc, systemd and upstart all supported), this is a problem solved. Ooh, that sounds like something that my team would like to use. Please could you point to more information about this tool? Sure! Note that I wrote it a few months ago, so it may have a bit outdated information (like, there are bugs in the scripts in this blog entry which have been fixed since), but the main principle remains: http://thomas.goirand.fr/blog/?p=212 Otherwise, just download any Kilo server source package from Debian or Ubuntu, and you'll see it in action (for example, Cinder, Nova...). Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On Wed, Jun 03, 2015 at 04:30:17PM -0400, Sean Dague wrote: The closer we can get logic about what a service should look like on disk back into that service itself, the less work duplicated by any of the installers, and the more common OpenStack envs would be. The fact that every installer / package needs to copy in a bunch of etc files (because the python packages don't do it) has always seemed rather odd to me. I agree with this, and given the disparate and seemingly contradictory goals driving these discussions, I think it will be exceedingly difficult to make everyone happy. So here's my suggestion: 1. Maintain all important data for packaging as first-class members of the respective repositories. Initscripts, systemd service files, licensing (SPDX?), and so on should be in the master branch of each project. Just enough glue should be present such that functional packaging can be programmatically generated from HEAD with debdry or similar tooling, and this is how test jobs should operate (f.ex.: run debdry, mangle version number to something unique, build package in chroot or equivalent, store output for use in other testing). 2. Create a single repository, with one subdirectory per source package, in which overrides or patches to third-party packaging can be committed and used to trigger builds. This way OpenStack could - produce and consume its own package artifacts for testing changes of varying complexity - get early warning of changes which will break packaging, insanity in dependency graphs, and so on - be a central point of collaboration without introducing a bunch of repo duplication or bypass of code review Distributors could - clone the upstream repos and branch to add any special tweaks, then potentially run the same automation to get source/binary packages - collaborate upstream to fix things of common utility __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Hi Clint, Thanks for your contribution to this thread. On 06/04/2015 10:35 PM, Clint Adams wrote: On Wed, Jun 03, 2015 at 04:30:17PM -0400, Sean Dague wrote: The closer we can get logic about what a service should look like on disk back into that service itself, the less work duplicated by any of the installers, and the more common OpenStack envs would be. The fact that every installer / package needs to copy in a bunch of etc files (because the python packages don't do it) has always seemed rather odd to me. I agree with this, and given the disparate and seemingly contradictory goals driving these discussions, I think it will be exceedingly difficult to make everyone happy. I don't think anyone involved in the packaging of OpenStack has expressed disparity or contradiction. Quite the opposite: it is my strong opinion that all parties involved are on the same page. So here's my suggestion: 1. Maintain all important data for packaging as first-class members of the respective repositories. Initscripts, systemd service files, licensing (SPDX?), and so on should be in the master branch of each project. Are you here saying we should move startup scripts upstream, and not on the packaging repos? If so, that's a bad idea. Let me explain. The init scripts used to be hard to maintain because they were many, but since Debian Ubuntu are using automatic generation out of a tiny template (with sysv-rc, systemd and upstart all supported), this is a problem solved. If individual projects are getting into the business of publishing their own startup scripts, I doubt that we would even use them, because having a central place to patch all of the startup scripts at once (ie: openstack-pkg-tools) is much better than having to maintain each and every startup script by hand. As for the licensing, I agree here. I have multiple times express my opinion about it: the project as a hole needs to make some progress, as it's nearly impossible for downstream distributions to second guess who is the copyright holders (please pay attention: copyright holders and licensing are 2 separate things...). Though SPDX?!? Do you know the famous quote (Jay Pipes) Get your dirty XML out of my Json ? :) And there's also the fact that Debian uses a different parseable format. Not sure what the RPM folks use, but I suppose that's embedded in a .spec file. Also, all of OpenStack is mostly using the Apache-2.0 license, the licensing issues are with (build-)dependencies, and that, we can't solve it. Just enough glue should be present such that functional packaging can be programmatically generated from HEAD with debdry or similar tooling, and this is how test jobs should operate (f.ex.: run debdry, mangle version number to something unique, build package in chroot or equivalent, store output for use in other testing). I already use pkgos-debpypi (from openstack-pkg-tools) to automate a big chunk of the Python module packaging. Though the automated thing can only drive you so far. It wont fix missing dependencies in requirement.txt, wrong lower bounds, SSLv3 removal related issues, and all of this kind of issues which we constantly need to address to make sure all unit tests are passing. The thing is, unit functional tests in OpenStack are designed for Devstack. I supposed you already saw the It worked on Devstack XKCD t-shirt... Well, I'm sure all package maintainers of OpenStack have this in mind every day! :) As you are a DD, you know it too: getting a correct debian/copyright also represent some work, which cannot be automated, or the FTP masters will really hate you. :P Plus there's also the fact that sometimes, distributions don't use the matching versions. For example, Debian Jessie was released with Django 1.7 and OpenStack Icehouse, and I had to work on patching Horizon to make it work with it (at the time, Horizon didn't work with something higher than 1.6). The same thing happened with SQLAlchemy. This has slowed down a little bit over the years, but we also used to spend most of our time simply packaging new dependencies. We already have a big amount of redundancy (alembic vs sqlalchemy-migrate, WSGI frameworks, nose vs testr, pymysql vs mysqlclient vs mysql-python, you name it...). And each new project, with its specificities, bring new dependencies. I used to say that 80% of the packaging work is spent there: packaging new stuff. These days, it has a bit shifted to packaging oslo and client libs, which is a good thing (as they are respecting a standard, so we have less surprise). Though what actually represent an OpenStack release has grown bigger. After Jessie was released, uploading all of Kilo to Sid took me about 3 or 4 days, and maybe about the same amount of time to upload it to the official Jessie backports (all done in dependency order, with as few dependency breakage as possible...). I really hope that the effort on having a gate on the lower bounds of our
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
[purely outside-looking-in observation below...] On Wed, Jun 3, 2015 at 1:07 PM, Andreas Jaeger a...@suse.com wrote: On 06/03/2015 03:57 PM, James Page wrote: [...] After some discussion with Thomas on IRC, I think this is more than one effort; The skills and motivation for developers reviewing proposed packaging changes needs to be aligned IMO - so I think it makes sense to split the packaging teams between: Debian/Ubuntu + derivatives CentOS/Fedora/RHEL + derivatives So, would this be two packaging teams - on DEB team and one RPM team? How would those two teams collaborate - or is no collaboration needed? I think it would be beneficial to have a single team and single PTL with multiple sets of repos (since the idea of branching per distro seems to be universally non-favored). Having a single team to discuss the desired converged end-results seems like a win to me, then each group goes off to implement with as much or as little shared process as can be worked out. Again, from the outside looking in it feels like if there are distinct teams that we'll repeat the past efforts to converge any practices and users will continue to have arbitrarily different (from their point of view) deployments. This is one reason, btw, that we are starting to work toward pushing some default configuration practices back into projects out of places like DevStack where they have become de facto defaults. hopefully upstream practices will then trickle back down into actual use. dt -- Dean Troyer dtro...@gmail.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 12:08 PM, Allison Randal wrote: On 06/03/2015 07:22 AM, Thomas Goirand wrote: However, talking with James Page (from Canonical, head of their server team which does the OpenStack packaging), we believe it's best if we had 2 different distinct teams: one for Fedora/SuSe/everything-rpm, and one for Debian based distribution. We could try to work as a single entity (RPM + deb teams), but rpm+yum and dpkg+apt are 2 distinct worlds which have very few common attributes. So even if it may socially be nice, it's not the right technical decision. Taking a step back, even though the tooling and packaging formats are different, it is a massive benefit to OpenStack and to operators if the end result of installing OpenStack packages on any distro is as similar as possible. To that end, this should be one unified packaging team focused on delivering a usable OpenStack through the distros. So wouldn't that be more of an arguement to move as much of the installation logic back into the python packages as possible. So that pip install nova was a thing that you could do, and get reasonable results, and then the packaging teams would work around bundling that and handling dependencies sanely for their platforms. The closer we can get logic about what a service should look like on disk back into that service itself, the less work duplicated by any of the installers, and the more common OpenStack envs would be. The fact that every installer / package needs to copy in a bunch of etc files (because the python packages don't do it) has always seemed rather odd to me. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-06-03 20:15:05 +0200 (+0200), Andreas Jaeger wrote: [...] You could still have one shared repository with the understanding of who approves what. Working on one repo makes it easier to see what the other team does. [...] For that matter, if different distros used different branches within a repo, those branches could have their own distinct core teams. The ACLs for that are straightforward. Makes it somewhat easier to propose/cherry-pick common changes across multiple branches where similarities exist, and to branch new distros from existing ones as templates if needed. Not saying those are necessarily good reasons, just potential options to consider. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 08:07 PM, Andreas Jaeger wrote: On 06/03/2015 03:57 PM, James Page wrote: [...] After some discussion with Thomas on IRC, I think this is more than one effort; The skills and motivation for developers reviewing proposed packaging changes needs to be aligned IMO - so I think it makes sense to split the packaging teams between: Debian/Ubuntu + derivatives CentOS/Fedora/RHEL + derivatives So, would this be two packaging teams - on DEB team and one RPM team? Yes. How would those two teams collaborate - or is no collaboration needed? I don't see what there would be collaboration on. Even the thing we work on is called differently (ie: specs in Red Hat, and source packages in Debian/Ubuntu). Though if there are things on which we can converge (like package names maybe?), it would be a good thing for final users. How are you handling Debian and Ubuntu differences? There's not so much difference already on the Python module dependencies, as Ubuntu imports packages from Debian. They imported 100% of my work for Juno, then diverged for Kilo since I didn't upload (as Jessie was frozen). Now that's the only difference we need to fix, and James is already working on that. We already work on some packages together (like James Page and myself are co-maintaining rabbitmq-server (he did most of it, shame on me)). As for the core OpenStack packages, what I wrote works already on Ubuntu if you rebuild the package, though it may need a few fix here and there for version dependencies (as Jessie was frozen after Trusty), but that's easy to fix. The harder part will be for Neutron and Nova. In case we don't agree at all, we can still use dpkg-vendors --derive-from ubuntu to check on which distribution we are, and do actions from there. There's already such mechanism in the debian/rules file of the Debian Nova package for example (to check on the group name for libvirt which differs in Debian Ubuntu). It's not in the Ubuntu version of the package, but I'm sure that it will soon land there! :) The only thing we can't fix will be the Maintainer: and Uploaders: field, but even that, maybe we can find a way so we don't have to fix it either. So in a smaller amount of words: I'm sure we can find a way to reduce our differences to the absolute zero on the source package level, even though the result may look different thanks to some conditionals where it can't be avoided. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 03:31 PM, Haïkel wrote: 2015-06-03 23:41 GMT+02:00 Allison Randal alli...@lohutok.net: I have to disagree on that point, integration with underlying OS and low-level services is important. If that integration doesn't exists, it's off-loaded to the operators. So downstream packages bring more value than pip deployment, as it will pull dependencies (not just things from PyPI), working combination with underlying OS components etc. Packages could be used in a variety of different configurations, even ones we didn't expect. Any sensitive scenario that we can't support is likely to be a packaging bug for me. In some cases, it makes sense for fine-tuning, but generally, you just want to get things work and tweak your configuration. Oh, yeah, I'm making an implicit distinction here that should be explicit. There are two different types of dependencies for an OpenStack service: - system dependencies that have to be installed on the same machine as the service, like a Python library used within the Nova code - service dependencies like a database, message queue, or other OpenStack service, which might be running on the same machine or might be running on a completely different machine System dependencies should be installed by the distro packaging, and you're right, distro packaging has an advantage over pip for installing non-Python system dependencies. (Though, typically with pip you're using it over the top of some other system config management tool, so the disadvantage is easy enough to iron out.) Service dependencies is where the problems lie, because neither pip nor distro packaging were designed as orchestration tools. Both pip and distro packaging should be consumable by any set of config management/orchestration tools, which means just install the software with minimal configuration. +1 As a matter of fact, I prefer that packages to be as agnostic as possible about the deployment and let that work to the orchestration tool. I'm a big fan of using the right tool for the job. Allison __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Hi Haikel! On 06/03/2015 06:28 PM, Haïkel wrote: First, we all agree that we should move packaging recipes (to use a neutral term) and reviewing to upstream gerrit. That should *NOT* be delayed. We (RDO) are even willing to transfer full control of the openstack-packages namespace on github. Thanks a lot! :) FYI, I'm also for having a separate namespace, just because adding more than 150 Git repositories at once in /openstack will be a huge mess. I don't really mind if it's called /openstack-packages or /openstack-pkg-deb or anything else, as long as we have consistency. Then, about the infra/tooling things, it looks like a misunderstanding. If we don't find an agreement on these topics, it's perfectly fine and should not prevent moving to upstream gerrit So let's break the discussion in two parts. 1. upstream gerrit shared by everyone and get this started asap 2. continue discussion about infra/tooling within the new project, without presumin the outcome. Does it look like a good compromise to you? Regards, H. I agree with what you wrote above. The tooling shouldn't be a blocker for the move to upstream Gerrit. We should decide on the tooling out of technically sound decisions that fits a given distribution. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 09:21 PM, Dean Troyer wrote: I think it would be beneficial to have a single team and single PTL with multiple sets of repos This isn't the direction we're taking, sorry. Yes, we can try to work as much as possible together, and try to have consistency across distributions if possible, but we're still completely different distribution types, with different tooling, package names, practices, policies, etc. On 06/03/2015 06:08 PM, Allison Randal wrote: it is a massive benefit to OpenStack and to operators if the end result of installing OpenStack packages on any distro is as similar as possible. To that end, this should be one unified packaging team focused on delivering a usable OpenStack through the distros. I'm afraid an utopia like this may lead to unpractical organization. Federating Debian + Ubuntu + MOS on one side, then all RPM based things on another is already a huge win. Don't ask for too much at once, and let us try to achieve some result first. On 06/03/2015 10:30 PM, Sean Dague wrote: So wouldn't that be more of an arguement to move as much of the installation logic back into the python packages as possible. So that pip install nova was a thing that you could do, and get reasonable results, and then the packaging teams would work around bundling that and handling dependencies sanely for their platforms. Well, not pip. No pip involved in packaging at all. python setup.py is what we do. And yes, PBR does wonderful things for that. But packaging isn't just only about that. It's also having the logrotate done right, cron jobs installed correctly, startup scripts (in Debian Ubuntu we support all of the 3 init systems: sysv-rc, upstart, systemd), bash completion, sudoers files, manpages, docs in place. All of what I just wrote is distribution specific, and cannot be offloaded to setuptools or PBR. The fact that every installer / package needs to copy in a bunch of etc files (because the python packages don't do it) has always seemed rather odd to me. The last time I saw upstream OpenStack trying to address it, we had configuration files copied on /usr/etc/neutron !!! :) Seriously, RPM and DEB have different ways of handling configuration files anyway (.rpmnew anyone? Is that configuration marked as CONFFILE debian lovers?). I'd love if we had a single distribution everyone would work on, but that's simply not the case, and it wont change tomorrow. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 01:30 PM, Sean Dague wrote: So wouldn't that be more of an arguement to move as much of the installation logic back into the python packages as possible. So that pip install nova was a thing that you could do, and get reasonable results, and then the packaging teams would work around bundling that and handling dependencies sanely for their platforms. The closer we can get logic about what a service should look like on disk back into that service itself, the less work duplicated by any of the installers, and the more common OpenStack envs would be. The fact that every installer / package needs to copy in a bunch of etc files (because the python packages don't do it) has always seemed rather odd to me. TBH, I don't think pip or distro packaging are ever going to be the right answer for fully configuring an OpenStack cloud. Because, there is no one true cloud, there are a variety of different configurations and combinations depending on whether you're in a dev/test scenario, running a private cloud, a public cloud, how many machines you're deploying to, what services you want to run on which machines, what your underlying network looks like, etc, etc... Having pip or distro packaging that's very opinionated about configuring a large set of related services is worse than useless when it's fighting against the configuration you need. It's on the order of installing the nginx package and finding that apt has set up a Wordpress instance and database you didn't want or need. Operator's nightmare. Both pip and distro packaging should be consumable by any set of config management/orchestration tools, which means just install the software with minimal configuration. Allison __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
2015-06-03 23:41 GMT+02:00 Allison Randal alli...@lohutok.net: TBH, I don't think pip or distro packaging are ever going to be the right answer for fully configuring an OpenStack cloud. Because, there is no one true cloud, there are a variety of different configurations and combinations depending on whether you're in a dev/test scenario, running a private cloud, a public cloud, how many machines you're deploying to, what services you want to run on which machines, what your underlying network looks like, etc, etc... I have to disagree on that point, integration with underlying OS and low-level services is important. If that integration doesn't exists, it's off-loaded to the operators. So downstream packages bring more value than pip deployment, as it will pull dependencies (not just things from PyPI), working combination with underlying OS components etc. Packages could be used in a variety of different configurations, even ones we didn't expect. Any sensitive scenario that we can't support is likely to be a packaging bug for me. In some cases, it makes sense for fine-tuning, but generally, you just want to get things work and tweak your configuration. Having pip or distro packaging that's very opinionated about configuring a large set of related services is worse than useless when it's fighting against the configuration you need. It's on the order of installing the nginx package and finding that apt has set up a Wordpress instance and database you didn't want or need. Operator's nightmare. Both pip and distro packaging should be consumable by any set of config management/orchestration tools, which means just install the software with minimal configuration. +1 As a matter of fact, I prefer that packages to be as agnostic as possible about the deployment and let that work to the orchestration tool. H. Allison __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-06-04 02:01:25 +0200 (+0200), Thomas Goirand wrote: FYI, I'm also for having a separate namespace, just because adding more than 150 Git repositories at once in /openstack will be a huge mess. [...] Simply from an infra standpoint there's no real distinction. We only ended up with that extra level of namespaces originally so we could match repo names up to the org structure imposed by Github. I and many others would love to just get rid of our current git namespaces entirely, so I'd prefer not to go adding new ones. A repo is a repo from the standpoint of Gerrit and the rest of our infrastructure, and the part of the name before the / is mostly arbitrary (and becoming more and more so with the Big Tent). -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 08:15 PM, Andreas Jaeger wrote: On 06/03/2015 04:22 PM, Thomas Goirand wrote: We could try to work as a single entity (RPM + deb teams), but rpm+yum and dpkg+apt are 2 distinct worlds which have very few common attributes. So even if it may socially be nice, it's not the right technical decision. You could still have one shared repository with the understanding of who approves what. Working on one repo makes it easier to see what the other team does. If you want to unify packaging - there are always arbitrary choices you can make - it's better to have a common team. I don't want to unify Red Hat and Debian (and its derivative) world, no. This isn't what this proposal is about. If you want to work on this yourself, I welcome you to do so, but so far, no party involved wants to work on this at the level you're discussing. All of the other Git repositories, with anyone OpenStack upstream, will move to upstream Gerrit. So, all of python-*client, python-oslo.*, python-xstatic-*, python-migrate, etc. I haven't yet counted how many repositories that represent, but that's a lot. Maybe between 150 to 200, and of course, growing as we constantly add new packages. Do you really need separate repositories for each package? Yes. Why not have one repository with several directories in it? Because that's not how git-buildpackage / sbuild works. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/02/2015 10:40 PM, Matthew Thode wrote: On 06/02/2015 05:41 PM, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. In general, I think there is a lot of support for a packaging effort in OpenStack. The discussion here has been great; we need to answer a few questions, get some decisions written down, and make sure we have agreement. Here's what we need to know: 1) Is this one or more than one horizontal effort? In other words, do we think the idea of having a single packaging project/team with collaboration among distros is going to work? Or should we look at it more like the deployment projects where we have puppet and chef as top level OpenStack projects? Either way is fine, and regardless, we need to answer the next questions: 2) What's the collaboration plan? How will different distros collaborate with each other, if at all? What things are important to standardize on, what aren't and how do we support them all. 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. 4) Who is on the team(s)? Who is interested in the overall effort? Who is signing up for distro-specific work? Who will be the initial PTL? I think if the discussion here can answer those questions, you should update the governance repo change with that information, we can get all the participants to ack that, and the TC will be able to act. Thanks again for driving this. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Gentoo packages from source client side, don't think this effects us. Possibly, and that's definitely a legit answer. I think in the deb packaging effort the primary desire is that package build files would be in Gerrit to encourage collaboration in the wider community. So an openstack/ebuild-packaging that was the git tree with the ebuilds could be a thing if it was a thing you wanted. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 04:15 PM, Derek Higgins wrote: o Tools to build packages in CI jobs should provide a consistent interface regardless of packaging being built Sure, we can have *some* of the tooling converging. But I don't see Debian/Ubuntu using anything else than git-buildpackage and sbuild (as this is what is used everywhere in Debian and Ubuntu), and I don't see how RPM guys could be using that either. 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. Assuming you mean git repositories ? I think anything under the openstack (or stackforge) umbrella Just to make things more clear... Here, we're talking about the /openstack namespace, which is why the TC is involved. Otherwise, pushing to /stackforge wouldn't require the blessing of the TC. If you meant package repositories I think none is a fine answer for the moment but if there is an appetite for them then I think what would eventually make most sense are repositories for master branches along with supported stable branches. This may differ between packaging formats and what their teams are prepared to support. As I wrote earlier, we can't technically avoid to have packages stored in upstream infra, because of build-dependency chains. Publishing the resulting packages publicly is another story, which we may decide later on. To me, this really is a tiny small implementation detail, as what counts anyway, is having stable packages available on distribution repositories, which means publishing stable package repositories publicly makes very little sense. I agree that publishing master/trunk could be a lot of fun though! Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 02/06/15 23:41, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. In general, I think there is a lot of support for a packaging effort in OpenStack. The discussion here has been great; we need to answer a few questions, get some decisions written down, and make sure we have agreement. Here's what we need to know: 1) Is this one or more than one horizontal effort? In other words, do we think the idea of having a single packaging project/team with collaboration among distros is going to work? Or should we look at it more like the deployment projects where we have puppet and chef as top level OpenStack projects? As far as packaging goes Id imaging the teams will be split into groups of people who are interested into specific packaging formats (or perhaps distro), these people would be responsible for package updates, reviews, etc... On the specifics of the packaging details, collaboration between these groups should be encouraged but not enforced. I would hope that this means we would find the places where packaging details can converge while staying within the constraints of distro recommendations. Either way is fine, and regardless, we need to answer the next questions: 2) What's the collaboration plan? How will different distros collaborate with each other, if at all? What things are important to standardize on, what aren't and how do we support them all. Collaboration between these groups is important in order to keep a few things consistent o package repository naming, we should all agree on a naming scheme for the packaging repositories to avoid situations where we have rpm-nova and deb-compute o Tools to build packages in CI jobs should provide a consistent interface regardless of packaging being built 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. Assuming you mean git repositories ? I think anything under the openstack(or stackforge) umbrella is fair game along with anything in the global-requirements file. If you meant package repositories I think none is a fine answer for the moment but if there is an appetite for them then I think what would eventually make most sense are repositories for master branches along with supported stable branches. This may differ between packaging formats and what their teams are prepared to support. 4) Who is on the team(s)? Who is interested in the overall effort? Who is signing up for distro-specific work? Who will be the initial PTL? From the RDO point of view we are doing the trunk chasing work already downstream. If we were to shift this packaging upstream of RDO I would imagine we would just switch the gerrit we are submitting too. I don't speak for RDO but of the people I spoke too I didn't hear any resistance to this idea. I think if the discussion here can answer those questions, you should update the governance repo change with that information, we can get all the participants to ack that, and the TC will be able to act. Great and thanks, Derek. Thanks again for driving this. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 06:47 AM, Sean Dague wrote: On 06/02/2015 10:40 PM, Matthew Thode wrote: On 06/02/2015 05:41 PM, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. In general, I think there is a lot of support for a packaging effort in OpenStack. The discussion here has been great; we need to answer a few questions, get some decisions written down, and make sure we have agreement. Here's what we need to know: 1) Is this one or more than one horizontal effort? In other words, do we think the idea of having a single packaging project/team with collaboration among distros is going to work? Or should we look at it more like the deployment projects where we have puppet and chef as top level OpenStack projects? Either way is fine, and regardless, we need to answer the next questions: 2) What's the collaboration plan? How will different distros collaborate with each other, if at all? What things are important to standardize on, what aren't and how do we support them all. 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. 4) Who is on the team(s)? Who is interested in the overall effort? Who is signing up for distro-specific work? Who will be the initial PTL? I think if the discussion here can answer those questions, you should update the governance repo change with that information, we can get all the participants to ack that, and the TC will be able to act. Thanks again for driving this. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Gentoo packages from source client side, don't think this effects us. Possibly, and that's definitely a legit answer. I think in the deb packaging effort the primary desire is that package build files would be in Gerrit to encourage collaboration in the wider community. So an openstack/ebuild-packaging that was the git tree with the ebuilds could be a thing if it was a thing you wanted. -Sean ya, that might be able to work, specifically with the package mapping proposal (we call oslo.messaging dev-python/oslo-messaging). We are already close as a distro to switching to git for our package repos, using cvs now. Maintenance of the ebuilds primarily consists of updating the dependencies between versions and maybe updating some tests (we allow end users to test before install of a package). We could even generate a deb or rpm from the ebuild too :P -- -- Matthew Thode (prometheanfire) signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi James On 02/06/15 23:41, James E. Blair wrote: This came up at the TC meeting today, and I volunteered to provide an update from the discussion. Thankyou - much appreciated. In general, I think there is a lot of support for a packaging effort in OpenStack. The discussion here has been great; we need to answer a few questions, get some decisions written down, and make sure we have agreement. Here's what we need to know: 1) Is this one or more than one horizontal effort? In other words, do we think the idea of having a single packaging project/team with collaboration among distros is going to work? Or should we look at it more like the deployment projects where we have puppet and chef as top level OpenStack projects? After some discussion with Thomas on IRC, I think this is more than one effort; The skills and motivation for developers reviewing proposed packaging changes needs to be aligned IMO - so I think it makes sense to split the packaging teams between: Debian/Ubuntu + derivatives CentOS/Fedora/RHEL + derivatives Either way is fine, and regardless, we need to answer the next questions: 2) What's the collaboration plan? How will different distros collaborate with each other, if at all? What things are important to standardize on, what aren't and how do we support them all. For Debian/Ubuntu, Thomas and I are already working on re-aligning packaging as much as possible for the Liberty cycle; to start off with this will be the dependency chain, but we've also agreed to look at the core openstack packages as well, which is where we have the greatest delta today. We will have to come up with some sort of smart way to continue to manage that delta going forward - we've had quite black and white opinions in the past as to what should be in the core packages and what should not. That said, we want to enable wider collaboration as well. By aligning branches for packaging against OpenStack releases, rather than Debian or Ubuntu releases, I think we can gain the maximum collaboration between Debian and Ubuntu, irrespective of which OpenStack version is being shipped by each distro. 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. I think Thomas has the definitive list of existing git repositories in Debian that we can use for the majority of repos - but I'd like to ensure we get the branches setup right on the openstack projects themselves to represent the simpler Ubuntu base packaging and the more complex Debian packaging. Thomas and I can work on the details of that. 4) Who is on the team(s)? Who is interested in the overall effort? Who is signing up for distro-specific work? Who will be the initial PTL? I'd like the members of my team who work on OpenStack packaging at Canonical to be part of the Debian/Ubuntu development team; that would include myself, Chuck Short and Corey Bryant. As Thomas is already taking a lead and has control of the majority of repository sources, I'd be happy for him to be the initial PTL; however I would like to see the PTL role switch each cycle so as to not overburden one individual, and to make sure that the team enjoys the diversity of technical leadership and objectives of each distro. I think if the discussion here can answer those questions, you should update the governance repo change with that information, we can get alhat.l the participants to ack that, and the TC will be able to act. Regards James - -- James Page Ubuntu and Debian Developer james.p...@ubuntu.com jamesp...@debian.org -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBCAAGBQJVbweuAAoJEL/srsug59jDRnEP/A6avj+hGBo46Y8H5K3LkEjn YgUteWnHs0QsrOSO5zZBdx/xzAK+ADbrOL4hQ/8vGoBhzy3MhQIPXbWemrowE+CK h3x71xYlSUzzxIuvLYmt+Gy9wsB/K5KEocB7hmgOL6lKRC1QyA0RF6RFEZ7HMbZ/ DydeeK0c4GW5mtrsZa708pVoWHsfcRpGUmX+80iXT4faREHQwTyscG/zb/xUaUhc yBd47tIagcs28nuy7xOENiWwb++ydgbDtzg6OOWa48Eb1Jtskeh6cyiW4Yk8G8mS OsOXZtRVpRpYAfu6dH8jg6k24I6oQBhPbqxz0bj4lI6eRvSPC+1fT8IvDHobisE/ rB2I51QXgBfeSFeOmZi2gP3lptXvA7cnY6z7L66BdVzooVbMJuLlUG50G4XXI4qt XOhb8c+Gk0DpMtMq34HsBNN512TeCIqfWBbo+ZwKHHEGET/5GrWxuiFaGg9nPvQW z/L88ew8pswulI64rxu7FKbokB245GNNutB6/nJY+YhqdrKR7ojROcXcgT1CYX6p TzyuahaepW+M6N0Xs1Gh+YmacxnyJDFNZQyM7FoHiUc4+Wwp/DTmxBS4+aoVvedP 3nJ51ueBoXu5a5sSSxNfwWH+mvVtwpoKi5KYR8FWaAG0z0x7Wf0WEKtR+TM4ixXp NNpLGulILZqefi6fyZ+w =NuyQ -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi James B., Thanks for this reply. As you asked for ACK from all parts, my words will be very much like the ones of James P. (I've just read his message, and I'm jealous of his nice native-English wording...:)). On 06/03/2015 12:41 AM, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. Thanks. In general, I think there is a lot of support for a packaging effort in OpenStack. The discussion here has been great; we need to answer a few questions, get some decisions written down, and make sure we have agreement. Here's what we need to know: 1) Is this one or more than one horizontal effort? In other words, do we think the idea of having a single packaging project/team with collaboration among distros is going to work? Or should we look at it more like the deployment projects where we have puppet and chef as top level OpenStack projects? I don't really know about the puppet project, so I don't know what you refer to here. However, talking with James Page (from Canonical, head of their server team which does the OpenStack packaging), we believe it's best if we had 2 different distinct teams: one for Fedora/SuSe/everything-rpm, and one for Debian based distribution. We could try to work as a single entity (RPM + deb teams), but rpm+yum and dpkg+apt are 2 distinct worlds which have very few common attributes. So even if it may socially be nice, it's not the right technical decision. At least, during the summit, what we agreed on is that we don't want Debian/Ubuntu guys having the rights to core-review RPM packaging and vice-versa. So the core reviewer lists will have to be separated. The list of repositories will also have to be split, because repositories must match package names, and we have different naming (and even naming policies). Plus the ACLs will be better managed on a per-repo basis than on a per-branch one. It would also maybe make sense to have separate PTLs too (though we're open to other views, if it's easier to have a single one for the rest of the community). Either way is fine, and regardless, we need to answer the next questions: 2) What's the collaboration plan? How will different distros collaborate with each other, if at all? What things are important to standardize on, what aren't and how do we support them all. I can answer only for the Debian/Ubuntu part, and I'll let the RPM world guys reply from themselves First, we already worked together between Debian and Ubuntu. All of Juno in Ubuntu was using Python packages I worked on (in fact, all of OpenStack but the core packages). We have the intention to at least merge all of our efforts on everything but the core packages first, then see on case by case basis how we can merge all packaging for the more complex packages. Nova and Neutron packages have unfortunately diverged over the years, so we have to be extra careful on how this is technically going to happen, without breaking any distro. But I'm confident we'll succeed. What sparked this, is that during the summit, Mark Shuttleworth told me he was supportive of more collaboration between Debian Ubuntu. Just 5 minutes after his words, I (very fortunately) bumped into James Page, and we decided to push everything to /stackforge, then try to merge all of our source packages. Then, when discussing the mater with others, I heard sentences like you should push this into the /openstack namespace to make it more big-tent-ish, on which I agree. So there is at least a strong will to maintain OpenStack packages for Debian and Ubuntu collectively. This means it would be both James Page team (for Canonical) and myself (working in Debian). I hope to push everyone else in Mirantis who works on MOS to also do packaging on upstream Gerrit, at least for the Debian/Ubuntu part. So that is 3 OpenStack distributions that will work as one. Also, I have to say that the merging effort between Debian and Ubuntu has already started. 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. Currently, OpenStack in Debian represents 237 packages, which are all listed there: https://qa.debian.org/developer.php?login=openstack-de...@lists.alioth.debian.org Some of the more generic ones will be moved to do package maintenance under the DPMT (Debian Python Module Team). I'm thinking for example about python-nose-parametrized, python-nose-timer, python-termcolor, python-termstyle, python-rednose, python-jingo, python-couleur, python-croniter, python-nosehtmloutput, python-nose-exclude, python-mockito... This kind of packages. That's maybe 20 to 30 packages to move there. However, I am waiting for the DPMT to finish its migration from SVN to Git, as I don't really want to jump 20 years in the past. These packages can
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
i On 06/03/2015 12:41 AM, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. I've just read the IRC logs. And there's one thing I would like to make super clear. We, ie: Debian Ubuntu folks, are very much clear on what we want to achieve. The project has been maturing in our heads for like more than 2 years. We would like that ultimately, only a single set of packages Git repositories exist. We already worked on *some* convergence during the last years, but now we want a *full* alignment. We're not 100% sure how the implementation details will look like for the core packages (like about using the Debconf interface for configuring packages), but it will eventually happen. For all the rest (ie: Python module packaging), which represent the biggest work, we're already converging and this has zero controversy. Now, the Fedora/RDO/Suse people jumped on the idea to push packaging on the upstream infra. Great. That's socially tempting. But technically, I don't really see the point, apart from some of the infra tooling (super cool if what Paul Belanger does works for both Deb+RPM). Finally, indeed, this is not totally baked. But let's please not delay the Debian+Ubuntu upstream Gerrit collaboration part because of it. We would like to get started, and for the moment, nobody is approving the /stackforge/deb-openstack-pkg-tools [1] new repository because we're waiting on the TC decision. Cheers, Thomas Goirand (zigo) [1] https://review.openstack.org/#/c/185164/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 07:22 AM, Thomas Goirand wrote: However, talking with James Page (from Canonical, head of their server team which does the OpenStack packaging), we believe it's best if we had 2 different distinct teams: one for Fedora/SuSe/everything-rpm, and one for Debian based distribution. We could try to work as a single entity (RPM + deb teams), but rpm+yum and dpkg+apt are 2 distinct worlds which have very few common attributes. So even if it may socially be nice, it's not the right technical decision. Taking a step back, even though the tooling and packaging formats are different, it is a massive benefit to OpenStack and to operators if the end result of installing OpenStack packages on any distro is as similar as possible. To that end, this should be one unified packaging team focused on delivering a usable OpenStack through the distros. Allison __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
2015-06-03 17:23 GMT+02:00 Thomas Goirand z...@debian.org: i On 06/03/2015 12:41 AM, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. I've just read the IRC logs. And there's one thing I would like to make super clear. I still haven't read the logs as we had our post-mortem meeting today, but I'll try to address your points. We, ie: Debian Ubuntu folks, are very much clear on what we want to achieve. The project has been maturing in our heads for like more than 2 years. We would like that ultimately, only a single set of packages Git repositories exist. We already worked on *some* convergence during the last years, but now we want a *full* alignment. We're not 100% sure how the implementation details will look like for the core packages (like about using the Debconf interface for configuring packages), but it will eventually happen. For all the rest (ie: Python module packaging), which represent the biggest work, we're already converging and this has zero controversy. Now, the Fedora/RDO/Suse people jumped on the idea to push packaging on the upstream infra. Great. That's socially tempting. But technically, I don't really see the point, apart from some of the infra tooling (super cool if what Paul Belanger does works for both Deb+RPM). Finally, indeed, this is not totally baked. But let's please not delay the Debian+Ubuntu upstream Gerrit collaboration part because of it. We would like to get started, and for the moment, nobody is approving the /stackforge/deb-openstack-pkg-tools [1] new repository because we're waiting on the TC decision. First, we all agree that we should move packaging recipes (to use a neutral term) and reviewing to upstream gerrit. That should *NOT* be delayed. We (RDO) are even willing to transfer full control of the openstack-packages namespace on github. If you want to use another namespace, it's also fine with us. Then, about the infra/tooling things, it looks like a misunderstanding. If we don't find an agreement on these topics, it's perfectly fine and should not prevent moving to upstream gerrit So let's break the discussion in two parts. 1. upstream gerrit shared by everyone and get this started asap 2. continue discussion about infra/tooling within the new project, without presumin the outcome. Does it look like a good compromise to you? Regards, H. Cheers, Thomas Goirand (zigo) [1] https://review.openstack.org/#/c/185164/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 11:23 AM, Thomas Goirand wrote: i On 06/03/2015 12:41 AM, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. I've just read the IRC logs. And there's one thing I would like to make super clear. We, ie: Debian Ubuntu folks, are very much clear on what we want to achieve. The project has been maturing in our heads for like more than 2 years. We would like that ultimately, only a single set of packages Git repositories exist. We already worked on *some* convergence during the last years, but now we want a *full* alignment. We're not 100% sure how the implementation details will look like for the core packages (like about using the Debconf interface for configuring packages), but it will eventually happen. For all the rest (ie: Python module packaging), which represent the biggest work, we're already converging and this has zero controversy. Now, the Fedora/RDO/Suse people jumped on the idea to push packaging on the upstream infra. Great. That's socially tempting. But technically, I don't really see the point, apart from some of the infra tooling (super cool if what Paul Belanger does works for both Deb+RPM). Finally, indeed, this is not totally baked. But let's please not delay the Debian+Ubuntu upstream Gerrit collaboration part because of it. We would like to get started, and for the moment, nobody is approving the /stackforge/deb-openstack-pkg-tools [1] new repository because we're waiting on the TC decision. I would agree with not gating on stuff that -infra is working on too. If getting gerrit collaboration is useful to Debian / Ubuntu, simple gate-noop testing seems like a easy solution. I agree, anything we in -infra can to do help establish some base tooling (chroots for example) would be super awesome. However, I also have expectation for packagers to use their own toolchains if more convenient. Cheers, Thomas Goirand (zigo) [1] https://review.openstack.org/#/c/185164/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 04:22 PM, Thomas Goirand wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi James B., Thanks for this reply. As you asked for ACK from all parts, my words will be very much like the ones of James P. (I've just read his message, and I'm jealous of his nice native-English wording...:)). On 06/03/2015 12:41 AM, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. Thanks. In general, I think there is a lot of support for a packaging effort in OpenStack. The discussion here has been great; we need to answer a few questions, get some decisions written down, and make sure we have agreement. Here's what we need to know: 1) Is this one or more than one horizontal effort? In other words, do we think the idea of having a single packaging project/team with collaboration among distros is going to work? Or should we look at it more like the deployment projects where we have puppet and chef as top level OpenStack projects? I don't really know about the puppet project, so I don't know what you refer to here. However, talking with James Page (from Canonical, head of their server team which does the OpenStack packaging), we believe it's best if we had 2 different distinct teams: one for Fedora/SuSe/everything-rpm, and one for Debian based distribution. We could try to work as a single entity (RPM + deb teams), but rpm+yum and dpkg+apt are 2 distinct worlds which have very few common attributes. So even if it may socially be nice, it's not the right technical decision. You could still have one shared repository with the understanding of who approves what. Working on one repo makes it easier to see what the other team does. If you want to unify packaging - there are always arbitrary choices you can make - it's better to have a common team. At least, during the summit, what we agreed on is that we don't want Debian/Ubuntu guys having the rights to core-review RPM packaging and vice-versa. So the core reviewer lists will have to be separated. The list of repositories will also have to be split, because repositories must match package names, and we have different naming (and even naming policies). Plus the ACLs will be better managed on a per-repo basis than on a per-branch one. This could also be a social contract. You might want to share config files between distros, so having them in one repository could make life easier. On the other hand, it means you have to collaborate to not break each other... It would also maybe make sense to have separate PTLs too (though we're open to other views, if it's easier to have a single one for the rest of the community). Either way is fine, and regardless, we need to answer the next questions: 2) What's the collaboration plan? How will different distros collaborate with each other, if at all? What things are important to standardize on, what aren't and how do we support them all. I can answer only for the Debian/Ubuntu part, and I'll let the RPM world guys reply from themselves First, we already worked together between Debian and Ubuntu. All of Juno in Ubuntu was using Python packages I worked on (in fact, all of OpenStack but the core packages). We have the intention to at least merge all of our efforts on everything but the core packages first, then see on case by case basis how we can merge all packaging for the more complex packages. Nova and Neutron packages have unfortunately diverged over the years, so we have to be extra careful on how this is technically going to happen, without breaking any distro. But I'm confident we'll succeed. What sparked this, is that during the summit, Mark Shuttleworth told me he was supportive of more collaboration between Debian Ubuntu. Just 5 minutes after his words, I (very fortunately) bumped into James Page, and we decided to push everything to /stackforge, then try to merge all of our source packages. Then, when discussing the mater with others, I heard sentences like you should push this into the /openstack namespace to make it more big-tent-ish, on which I agree. So there is at least a strong will to maintain OpenStack packages for Debian and Ubuntu collectively. This means it would be both James Page team (for Canonical) and myself (working in Debian). I hope to push everyone else in Mirantis who works on MOS to also do packaging on upstream Gerrit, at least for the Debian/Ubuntu part. So that is 3 OpenStack distributions that will work as one. Also, I have to say that the merging effort between Debian and Ubuntu has already started. 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. Currently, OpenStack in Debian represents 237 packages, which are all listed there: https://qa.debian.org/developer.php?login=openstack-de...@lists.alioth.debian.org Some of the more
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/03/2015 03:57 PM, James Page wrote:
[...]
After some discussion with Thomas on IRC, I think this is more than
one effort; The skills and motivation for developers reviewing
proposed packaging changes needs to be aligned IMO - so I think it
makes sense to split the packaging teams between:
Debian/Ubuntu + derivatives
CentOS/Fedora/RHEL + derivatives
So, would this be two packaging teams - on DEB team and one RPM team?
How would those two teams collaborate - or is no collaboration needed?
Btw. I'd like to throw SUSE into the equation but since none of us
participated in the face to face meeting, we need some better
understanding how this could work.
How are you handling Debian and Ubuntu differences? How shall we handle
differences between RPM distros?
Andreas
--
Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton,
HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/01/2015 07:16 PM, Jeremy Stanley wrote: On 2015-06-01 14:55:06 +0200 (+0200), Thomas Goirand wrote: [...] So, should I start writing a script to build an image for package building (ie: an image with sbuild, git-buildpackage, and so on...)? [...] Probably what we'd want to do is something like debootstrap/rpmstrap FYI, I was the Debian package maintainer for rpmstrap, and I just gave up on it because it was not maintainable (ie: breaking constantly). Bootstraping an RPM distribution can simply be done with yum (which is why I maintain Yum in Debian). a chroot for each platform we want to build, then in each of them iterate through the packaging git repos and --download-only the build-deps listed therein. That will prime a local cache in each chroot and then it will get baked into that image. Ok, got the idea. Though what should be done is fill /var/cache/pbuilder/aptarchive rather than the host OS. I'll try to do with something of that sort. Later when a worker is booted from that image, the package build job just chroots into the appropriate filesystem subtree and has a warm cache available to it so it only needs to hopefully update at most the package list and maybe a handful of packages before starting to build whatever new package was requested. Ok. That's what sbuild will do if we fill /var/cache/pbuilder/aptarchive with relevant content. The good thing about this approach is that it can be added later, it doesn't need to be implemented on day one. Yeah. I'm currently working on a Jessie cloud image with an sbuild configuration, so that when Paul is ready, we can use that. How would a job get the latest version of a Git repository then? This still needs network, right? [...] The way our jobs work right now is that the workers start with a recent (generally no more than a day old) clone of all the Git repositories we maintain. It still has to hit the network to retrieve more recent Git refs, but this at least minimizes network interaction and significantly reduces the failure rate. That will be the little bit more tricky part. Some libraries are very small, and probably caching will not be useful (too much work when building the VM image). However, for big projects (nova, neutron, cinder...), then we'll have to do something for Git repository caching. Thanks for your input Jeremy, this is very valuable. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. In general, I think there is a lot of support for a packaging effort in OpenStack. The discussion here has been great; we need to answer a few questions, get some decisions written down, and make sure we have agreement. Here's what we need to know: 1) Is this one or more than one horizontal effort? In other words, do we think the idea of having a single packaging project/team with collaboration among distros is going to work? Or should we look at it more like the deployment projects where we have puppet and chef as top level OpenStack projects? Either way is fine, and regardless, we need to answer the next questions: 2) What's the collaboration plan? How will different distros collaborate with each other, if at all? What things are important to standardize on, what aren't and how do we support them all. 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. 4) Who is on the team(s)? Who is interested in the overall effort? Who is signing up for distro-specific work? Who will be the initial PTL? I think if the discussion here can answer those questions, you should update the governance repo change with that information, we can get all the participants to ack that, and the TC will be able to act. Thanks again for driving this. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/02/2015 05:41 PM, James E. Blair wrote: Hi, This came up at the TC meeting today, and I volunteered to provide an update from the discussion. In general, I think there is a lot of support for a packaging effort in OpenStack. The discussion here has been great; we need to answer a few questions, get some decisions written down, and make sure we have agreement. Here's what we need to know: 1) Is this one or more than one horizontal effort? In other words, do we think the idea of having a single packaging project/team with collaboration among distros is going to work? Or should we look at it more like the deployment projects where we have puppet and chef as top level OpenStack projects? Either way is fine, and regardless, we need to answer the next questions: 2) What's the collaboration plan? How will different distros collaborate with each other, if at all? What things are important to standardize on, what aren't and how do we support them all. 3) What are the plans for repositories and their contents? What repos will be created, and what will be in them. When will new ones be created, and is there any process around that. 4) Who is on the team(s)? Who is interested in the overall effort? Who is signing up for distro-specific work? Who will be the initial PTL? I think if the discussion here can answer those questions, you should update the governance repo change with that information, we can get all the participants to ack that, and the TC will be able to act. Thanks again for driving this. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Gentoo packages from source client side, don't think this effects us. -- -- Matthew Thode (prometheanfire) signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-06-02 12:39:30 + (+), Jeremy Stanley wrote: Well, my point is those repos are already cached on every worker in /opt/git (e.g., /opt/git/openstack/nova) and you can git clone, cp or rsync those into your package build chroot. Then git remote set-url, update and reset --hard to the branch you want [...] Better still, we also preinstall a /usr/zuul-env/bin/zuul-cloner utility on all the workers which understands how to find those local caches, update them and reset them to the change(s) being tested for you based on environment variables passed into the build from our CI. An example use can be seen at URL: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/diskimage-builder.yaml where diskimage-builder functional tests are integrating and validating potential changes from two repos (diskimage-builder and dib-utils). -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-06-02 09:02:42 +0200 (+0200), Thomas Goirand wrote: [...] That will be the little bit more tricky part. Some libraries are very small, and probably caching will not be useful (too much work when building the VM image). However, for big projects (nova, neutron, cinder...), then we'll have to do something for Git repository caching. [...] Well, my point is those repos are already cached on every worker in /opt/git (e.g., /opt/git/openstack/nova) and you can git clone, cp or rsync those into your package build chroot. Then git remote set-url, update and reset --hard to the branch you want so that you're just pulling the delta from the last few hours (up to a day's worth generally, if everything is working correctly). Right now all our job workers have basically current caches of all ~700 repos hosted in our Gerrit. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 06/02/2015 02:39 PM, Jeremy Stanley wrote: On 2015-06-02 09:02:42 +0200 (+0200), Thomas Goirand wrote: [...] That will be the little bit more tricky part. Some libraries are very small, and probably caching will not be useful (too much work when building the VM image). However, for big projects (nova, neutron, cinder...), then we'll have to do something for Git repository caching. [...] Well, my point is those repos are already cached on every worker in /opt/git (e.g., /opt/git/openstack/nova) and you can git clone, cp or rsync those into your package build chroot. Then git remote set-url, update and reset --hard to the branch you want so that you're just pulling the delta from the last few hours (up to a day's worth generally, if everything is working correctly). Right now all our job workers have basically current caches of all ~700 repos hosted in our Gerrit. Ah, super cool! :) Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 05/29/2015 11:03 PM, Jeremy Stanley wrote: On 2015-05-28 23:19:36 +0200 (+0200), Thomas Goirand wrote: [...] By the way, I was thinking about the sbuild package caching system, and thought: how about network mounting /var/cache/pbuilder/aptcache using something like Manila (or any other distributed filesystem)? Does infra have such tooling in place? We pre-cache resources onto the local filesystems of the images used to boot our job workers, and update those images daily. Oh! This would work very well indeed. Now I think I understand better how we should be doing things. So, should I start writing a script to build an image for package building (ie: an image with sbuild, git-buildpackage, and so on...)? What would be the distributed filesystem of choice in such a case? We have an AFS cell which we could consider using for this eventually. We're working on using it as a distributed backend for package mirrors, git repositories, documentation and potentially lots of other things. However, as we've seen repeatedly, any actions in a job which require network access (even locally to other servers in the same cloud provider/region) have a variable percentage of failures associated with network issues. The more you can avoid depending on network resources, the better off your jobs will be. How would a job get the latest version of a Git repository then? This still needs network, right? Also, could we setup approx somewhere? Or do we have Debian and Ubuntu mirrors available within infra? [...] There is a separate effort underway to maintain distributed multi-distro package mirrors in all our providers/regions like we already do for our PyPI mirrors. Niiice! As mentioned above, it will likely be updated in one place on a writeable AFS volume, automatically consistency-checked, and then released to the read-only volumes published from each of the mirror servers. This sounds like very well thought. Please let me know when this is done. Maybe we can later update httpredir.debian.org (the HTTP redirector from Debian) to point to the correct mirror location then. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 05/29/2015 11:37 AM, Derek Higgins wrote: Whats important I think is that we can change things to use sbuild without docker if that is what works best for you for debs. Ok. Though if we are to use delorean, we'll have to switch to that in upstream Debian as well, and I'm not sure if that's relevant and how it would be received by the Debian community. Probably we shouldn't play with fire and avoid doing this. By that, I mean I would like infra to use whatever tooling everyone uses in Debian, and for the moment, that's sbuild + git-buildpackage. I think the feature in delorean that is most useful is that it will continue to maintain a history of usable package repositories representing the openstack projects over time, for this we would need a long running instance, but that can happen outside of infra. As much as I understand, the packages could be kept using swift, and then we can use something like sftpcloudfs or something similar to have the repository useable by instances. So a long living instance wouldn't be necessary, even though that's one of the possible implementation. Once we have all of the packaging available in infra we can use any tool to build it as part of CI, my preference for delorean is because it would match how we would want to run a long running delorean server. What infra folks told me is that they prefer not to maintain long living instances. For us, there's no such a need anyway (especially if we can solve the sbuild package caching thing...), unless we get into the business of maintaining a buildd network, dak and wanabuild, but really, that'd be too much work and I'm not sure that'd be useful. I'd very much prefer investing such a time on adding PPA support to dak. All of this needs to be preceded by actually importing the packaging into review.openstack.org , so lets talk to infra first about how we should go about that and we can converge on processes secondary to that? I've added this first repository: https://review.openstack.org/#/c/185164/ though for the moment, it has received zero reviews. As you can see, I've used /stackforge/deb-* as prefix. This will be used by both Debian and Ubuntu (we already both use it in our packaging). Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-06-01 14:55:06 +0200 (+0200), Thomas Goirand wrote: [...] So, should I start writing a script to build an image for package building (ie: an image with sbuild, git-buildpackage, and so on...)? [...] Probably what we'd want to do is something like debootstrap/rpmstrap a chroot for each platform we want to build, then in each of them iterate through the packaging git repos and --download-only the build-deps listed therein. That will prime a local cache in each chroot and then it will get baked into that image. Later when a worker is booted from that image, the package build job just chroots into the appropriate filesystem subtree and has a warm cache available to it so it only needs to hopefully update at most the package list and maybe a handful of packages before starting to build whatever new package was requested. The good thing about this approach is that it can be added later, it doesn't need to be implemented on day one. If we design the jobs correctly they'll download whatever they need, and then we can just iterate on improving our caches to improve performance. How would a job get the latest version of a Git repository then? This still needs network, right? [...] The way our jobs work right now is that the workers start with a recent (generally no more than a day old) clone of all the Git repositories we maintain. It still has to hit the network to retrieve more recent Git refs, but this at least minimizes network interaction and significantly reduces the failure rate. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 28/05/15 22:09, Thomas Goirand wrote: On 05/28/2015 02:53 PM, Derek Higgins wrote: On 28/05/15 12:07, Jaume Devesa wrote: Hi Thomas, Delorean is a tool to build rpm packages from master branches (maybe any branch?) of OpenStack projects. Check out here: https://www.rdoproject.org/packaging/rdo-packaging.html#master-pkg-guide Following those instructions you'll notice that the rpm's are being built using rpmbuild inside a docker container, if expanding to add dep support this is where we could plug in sbuild. sbuild by itself already provides the single use trow-able chroot feature, with very effective back ends like AUFS or LVM snapshots. Adding docker would only have the bad effect to remove the package caching feature of sbuild, so it makes no sense to use it, as sbuild would constantly download from the internet instead of using its package cache. Also, it is my understanding that infra will not accept to use long-living VMs, and prefer to spawn new instances. In such a case, I don't see the point using docker which would be a useless layer. In fact, I was even thinking that in this case, sbuild wouldn't be required, and we could simply use mk-build-deps and git-buildpackage without even using sbuild. The same dependency resolver (ie: apt) would then be in use, just without the added sbuild layer. I used that already to automatically build backports, and it was really fast. Did I miss something here? Apart from the fact that Docker is trendy, what feature would it bring? The reason I choose docker was essentially for a chroot to build the packages in while having the various distro images easily available, other people have shown interest in using mock in the passed so we may switch to it at some stage in the future. Whats important I think is that we can change things to use sbuild without docker if that is what works best for you for debs. I think the feature in delorean that is most useful is that it will continue to maintain a history of usable package repositories representing the openstack projects over time, for this we would need a long running instance, but that can happen outside of infra. Once we have all of the packaging available in infra we can use any tool to build it as part of CI, my preference for delorean is because it would match how we would want to run a long running delorean server. All of this needs to be preceded by actually importing the packaging into review.openstack.org , so lets talk to infra first about how we should go about that and we can converge on processes secondary to that? I'm traveling for a lot of next week but would like to try and start working on importing things to gerrit soon, so will try and get some prep done over the next week to import the RDO packaging but in reality it will probably be the following week before its ready (unless of course somebody else wants todo it). By the way, one question: does Delorean use mock? We had the discussion during an internal meeting, and we were not sure about this... Nope, not using mock currently Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 28/05/15 20:58, Paul Belanger wrote:
On 05/27/2015 05:26 PM, Derek Higgins wrote:
On 27/05/15 09:14, Thomas Goirand wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi all,
tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the
overall
QA for packaging *and* upstream.
- - The goal is *not* to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)
Longer version:
Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.
During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.
A number of topics have been raised, which needs to be shared.
First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.
So here it is. Suse people (and other distributions), you're welcome to
join the effort.
- - Why doing this
It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
that we'd be a way more effective if we worked better together, on a
collaborative fashion using a review process like on upstream Gerrit.
But also, we'd like to welcome anyone, and especially the operation
folks, to contribute and give feedback. Using Gerrit is the obvious way
to give everyone a say on what we're implementing.
As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.
This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.
We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.
Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already *before*
this is implemented).
- - What it will *not* be
===
We do not have the intention (yet?) to publish the resulting packages
built on upstream infra. Yes, we will share the same Git repositories,
and yes, the infra will need to keep a copy of all builds (for example,
because core packages will need oslo.db to build and run unit tests).
But we will still upload on each distributions on separate repositories.
So published packages by the infra isn't currently discussed. We could
get to this topic once everything is implemented, which may be nice
(because we'd have packages following trunk), though please, refrain to
engage in this topic right now: having the implementation done is more
important for the moment. Let's try to stay on tracks and be
constructive.
- - Let's keep efficiency in mind
===
Over the last few years, I've been able to maintain all of OpenStack in
Debian with little to no external contribution. Let's hope that the
Gerrit workflow will not slow down too much the packaging work, even if
there's an unavoidable overhead. Hopefully, we can implement some
liberal ACL policies for the core reviewers so that the Gerrit workflow
don't slow down anyone too much. For example we may be able to create
new repositories very fast, and it may be possible to self-approve some
of the most trivial patches (for things like typo in a package
description, adding new debconf translations, and such obvious fixes, we
shouldn't waste our time).
There's a middle ground between the current system (ie: only write
access ACLs for
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 29/05/15 02:54, Steve Kowalik wrote: On 29/05/15 06:41, Haïkel wrote: Here's the main script to rebuild a RPM package. https://github.com/openstack-packages/delorean/blob/master/scripts/build_rpm.sh The script basically uses rpmbuild to build packages, we could have a build_deb.sh that uses sbuild and add dockerfiles for the Debian/Ubuntu supported releases. I have a preliminary patch locally that adds support for building for both Debian and Ubuntu. I will be applying some polish next week and then working with the Delorean guys to get it landed. Thanks Steve, looking forward to seeing it __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-05-28 22:45:37 + (+), Fox, Kevin M wrote: You could pass the cache through with a volume Yeah, from the what can we do with our current CI infrastructure? perspective, we would just need a way to identify what bits benefit from being cached for these particular builds and then we would bake them into our worker base images just like we do to pre-cache all sorts of other resources used by all our jobs. The trick is to first understand what mechanisms we already have in the OpenStack CI to handle performance and isolation, and then try not to propose new solutions that redundantly reimplement them. Hopefully as we get more project infrastructure team members involved with this we can weed out the unnecessary additional complexities which are being suggested in the initial brainstorming process. As for what docker buys you, it would allow a vm (or my desktop) running centos (as an example) to build packages for multiple distro's using the distro's own native tools. I see that as a plus. I think the question was really not what good is isolation? but rather, how is a container any better than a chroot for this purpose? -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-05-28 23:09:41 +0200 (+0200), Thomas Goirand wrote: [...] Also, it is my understanding that infra will not accept to use long-living VMs, and prefer to spawn new instances. [...] Right, after we run arbitrary user-submitted code on a server, we cease to be able to trust it and so immediately delete and replace it. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-05-29 10:37:43 +0100 (+0100), Derek Higgins wrote: [...] I think the feature in delorean that is most useful is that it will continue to maintain a history of usable package repositories representing the openstack projects over time, for this we would need a long running instance, but that can happen outside of infra. [...] It could potentially even happen in our infrastructure, as long as the package builds happen on different machines from the servers serving the results of those builds. We don't eschew all long-running hosts, just long-running job workers. However, whether OpenStack's project infrastructure wants to be responsible for serving package download repositories for things it doesn't use is a separate discussion entirely. To me, the main benefits are that you have somewhere to collaborate on packaging work with a code review system, and our CI can further assist you in that effort by providing feedback on whether or not a proposed packaging change is able to successfully build a package (and perhaps even whether that package can be installed and exercised in some arbitrary way). -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On Fri, May 29, 2015 at 1:48 PM Jeremy Stanley fu...@yuggoth.org wrote: On 2015-05-28 23:09:41 +0200 (+0200), Thomas Goirand wrote: Also, it is my understanding that infra will not accept to use long-living VMs, and prefer to spawn new instances. Right, after we run arbitrary user-submitted code on a server, we cease to be able to trust it and so immediately delete and replace it. I think is unnecessarily maximalist. Trust is not an all-or-nothing boolean flag: why can't you trust that server to do more work at the same level of trust and run another batch of user-submitted code? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-05-28 23:19:36 +0200 (+0200), Thomas Goirand wrote: [...] By the way, I was thinking about the sbuild package caching system, and thought: how about network mounting /var/cache/pbuilder/aptcache using something like Manila (or any other distributed filesystem)? Does infra have such tooling in place? We pre-cache resources onto the local filesystems of the images used to boot our job workers, and update those images daily. What would be the distributed filesystem of choice in such a case? We have an AFS cell which we could consider using for this eventually. We're working on using it as a distributed backend for package mirrors, git repositories, documentation and potentially lots of other things. However, as we've seen repeatedly, any actions in a job which require network access (even locally to other servers in the same cloud provider/region) have a variable percentage of failures associated with network issues. The more you can avoid depending on network resources, the better off your jobs will be. Also, could we setup approx somewhere? Or do we have Debian and Ubuntu mirrors available within infra? [...] There is a separate effort underway to maintain distributed multi-distro package mirrors in all our providers/regions like we already do for our PyPI mirrors. As mentioned above, it will likely be updated in one place on a writeable AFS volume, automatically consistency-checked, and then released to the read-only volumes published from each of the mirror servers. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 2015-05-29 21:01:23 + (+), Dmitry Borodaenko wrote: I think is unnecessarily maximalist. Trust is not an all-or-nothing boolean flag: why can't you trust that server to do more work at the same level of trust and run another batch of user-submitted code? Because it turns out to be a lot easier not to have to figure out what's safe for node reuse and what isn't. We used to draw the line between jobs which don't have root access and jobs which do, but that was not actually a completely sane distinction for a number of reasons and hampered flexibility of our job definitions as well. Also, if you turn over your workers fairly rapidly, you don't need to worry about care and feeding, package updates, et cetera. Back when we had long-running workers for things like unit test jobs, a substantial chunk of my time was spent hunting down problems with them becoming disconnected from their Jenkins masters, developing subtle issues which would cause them to start rapidly failing any job run on them, and so on. Our least available resource is people, so saving them from needing to make decisions is one of the best ways we can increase the performance of our community (even if it means reducing performance of the systems being used to support them). -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
You could pass the cache through with a volume As for what docker buys you, it would allow a vm (or my desktop) running centos (as an example) to build packages for multiple distro's using the distro's own native tools. I see that as a plus. Thanks, Kevin From: Thomas Goirand [z...@debian.org] Sent: Thursday, May 28, 2015 2:09 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project On 05/28/2015 02:53 PM, Derek Higgins wrote: On 28/05/15 12:07, Jaume Devesa wrote: Hi Thomas, Delorean is a tool to build rpm packages from master branches (maybe any branch?) of OpenStack projects. Check out here: https://www.rdoproject.org/packaging/rdo-packaging.html#master-pkg-guide Following those instructions you'll notice that the rpm's are being built using rpmbuild inside a docker container, if expanding to add dep support this is where we could plug in sbuild. sbuild by itself already provides the single use trow-able chroot feature, with very effective back ends like AUFS or LVM snapshots. Adding docker would only have the bad effect to remove the package caching feature of sbuild, so it makes no sense to use it, as sbuild would constantly download from the internet instead of using its package cache. Also, it is my understanding that infra will not accept to use long-living VMs, and prefer to spawn new instances. In such a case, I don't see the point using docker which would be a useless layer. In fact, I was even thinking that in this case, sbuild wouldn't be required, and we could simply use mk-build-deps and git-buildpackage without even using sbuild. The same dependency resolver (ie: apt) would then be in use, just without the added sbuild layer. I used that already to automatically build backports, and it was really fast. Did I miss something here? Apart from the fact that Docker is trendy, what feature would it bring? By the way, one question: does Delorean use mock? We had the discussion during an internal meeting, and we were not sure about this... Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 05/28/2015 04:32 PM, Haïkel wrote: 2015-05-28 21:58 GMT+02:00 Paul Belanger pabelan...@redhat.com: Personally, I'm a fan of mock. Is there plan to add support for it? Also, currently containers are not being used in -infra. Not saying it is a show stopper, but could see some initial planning that is required for it. Nothing prevents us to run mock within Delorean container but I think this would be an useless overhead, we already have (much better) isolation with Docker. Moreover, leveraging docker is currently an option by mock upstream. Because I don't know, is Fedora now using docker (containers) as official build environments? Or still using mock to create chroots? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 05/28/2015 09:58 PM, Paul Belanger wrote: Not sure I'm a fan of rdorpm, seems too specific to RDO and would not foster other people using the git repo for packaging. Personally, I simple say rpm- prefix, allowing for branches to be used for distro specific changes. I full agree with that. 5. Add deb support to delorean, I know of at least one person who has already explored this (steve cc'd), if delorean is too far off the path of what we want to achieve and there is a better tool then I'm open to change. Personally, I'm a fan of mock. Is there plan to add support for it? It is my understanding that the approach of mock is more like the one of sbuild, which basically is to maintain a chroot template which is duplicated before the build, then trashed after it, without added layer like Docker. What back-end does mock support? AUFS? LVM snapshot? tmpfs? Also, currently containers are not being used in -infra. Not saying it is a show stopper, but could see some initial planning that is required for it. Yup, exactly my thinking. Adding a docker container within a VM just because that's what the build tool uses seems like a not-needed layer. By the way, I was thinking about the sbuild package caching system, and thought: how about network mounting /var/cache/pbuilder/aptcache using something like Manila (or any other distributed filesystem)? Does infra have such tooling in place? What would be the distributed filesystem of choice in such a case? Also, could we setup approx somewhere? Or do we have Debian and Ubuntu mirrors available within infra? Maybe mirrors would be the best choice here if we have enough resources for it. FYI, only amd64 arch would take about 120GB, with a reasonable amount of daily downloads for updating Sid (ie: with all arch, the daily downloads are about 6 GB, but with amd64, this should drop significantly). These were stats that I read a year or 2 ago, though I don't think it has changed much over time. Thoughts about this anyone? Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
2015-05-28 21:58 GMT+02:00 Paul Belanger pabelan...@redhat.com: Personally, I'm a fan of mock. Is there plan to add support for it? Also, currently containers are not being used in -infra. Not saying it is a show stopper, but could see some initial planning that is required for it. Nothing prevents us to run mock within Delorean container but I think this would be an useless overhead, we already have (much better) isolation with Docker. Moreover, leveraging docker is currently an option by mock upstream. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
2015-05-28 10:40 GMT+02:00 Thomas Goirand z...@debian.org: I don't know delorean at all, but what should be kept in mind is that, for Debian and Ubuntu, we *must* use sbuild, which is what is used on the buildd networks. I also started working on openstack-pkg-tools to provide such sbuild based build env, so I'm not sure if we need to switch to Delorean. Could you point me to some documentation about it, so I can see by myself what Delorean is about? Delorean basically will retrieve upstream sources and packaging recipes (ie: spec + config files for RPM), and will run a script to build packages in a docker container. Here's the main script to rebuild a RPM package. https://github.com/openstack-packages/delorean/blob/master/scripts/build_rpm.sh The script basically uses rpmbuild to build packages, we could have a build_deb.sh that uses sbuild and add dockerfiles for the Debian/Ubuntu supported releases. Regards, H. Cheers, Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 05/27/2015 05:26 PM, Derek Higgins wrote:
On 27/05/15 09:14, Thomas Goirand wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi all,
tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the
overall
QA for packaging *and* upstream.
- - The goal is *not* to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)
Longer version:
Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.
During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.
A number of topics have been raised, which needs to be shared.
First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.
So here it is. Suse people (and other distributions), you're welcome to
join the effort.
- - Why doing this
It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
that we'd be a way more effective if we worked better together, on a
collaborative fashion using a review process like on upstream Gerrit.
But also, we'd like to welcome anyone, and especially the operation
folks, to contribute and give feedback. Using Gerrit is the obvious way
to give everyone a say on what we're implementing.
As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.
This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.
We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.
Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already *before*
this is implemented).
- - What it will *not* be
===
We do not have the intention (yet?) to publish the resulting packages
built on upstream infra. Yes, we will share the same Git repositories,
and yes, the infra will need to keep a copy of all builds (for example,
because core packages will need oslo.db to build and run unit tests).
But we will still upload on each distributions on separate repositories.
So published packages by the infra isn't currently discussed. We could
get to this topic once everything is implemented, which may be nice
(because we'd have packages following trunk), though please, refrain to
engage in this topic right now: having the implementation done is more
important for the moment. Let's try to stay on tracks and be
constructive.
- - Let's keep efficiency in mind
===
Over the last few years, I've been able to maintain all of OpenStack in
Debian with little to no external contribution. Let's hope that the
Gerrit workflow will not slow down too much the packaging work, even if
there's an unavoidable overhead. Hopefully, we can implement some
liberal ACL policies for the core reviewers so that the Gerrit workflow
don't slow down anyone too much. For example we may be able to create
new repositories very fast, and it may be possible to self-approve some
of the most trivial patches (for things like typo in a package
description, adding new debconf translations, and such obvious fixes, we
shouldn't waste our time).
There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever) and a
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
2015-05-27 23:26 GMT+02:00 Derek Higgins der...@redhat.com:
On 27/05/15 09:14, Thomas Goirand wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi all,
tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the overall
QA for packaging *and* upstream.
- - The goal is *not* to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)
Longer version:
Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.
During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.
A number of topics have been raised, which needs to be shared.
First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.
So here it is. Suse people (and other distributions), you're welcome to
join the effort.
- - Why doing this
It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
that we'd be a way more effective if we worked better together, on a
collaborative fashion using a review process like on upstream Gerrit.
But also, we'd like to welcome anyone, and especially the operation
folks, to contribute and give feedback. Using Gerrit is the obvious way
to give everyone a say on what we're implementing.
As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.
This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.
We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.
Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already *before*
this is implemented).
- - What it will *not* be
===
We do not have the intention (yet?) to publish the resulting packages
built on upstream infra. Yes, we will share the same Git repositories,
and yes, the infra will need to keep a copy of all builds (for example,
because core packages will need oslo.db to build and run unit tests).
But we will still upload on each distributions on separate repositories.
So published packages by the infra isn't currently discussed. We could
get to this topic once everything is implemented, which may be nice
(because we'd have packages following trunk), though please, refrain to
engage in this topic right now: having the implementation done is more
important for the moment. Let's try to stay on tracks and be constructive.
- - Let's keep efficiency in mind
===
Over the last few years, I've been able to maintain all of OpenStack in
Debian with little to no external contribution. Let's hope that the
Gerrit workflow will not slow down too much the packaging work, even if
there's an unavoidable overhead. Hopefully, we can implement some
liberal ACL policies for the core reviewers so that the Gerrit workflow
don't slow down anyone too much. For example we may be able to create
new repositories very fast, and it may be possible to self-approve some
of the most trivial patches (for things like typo in a package
description, adding new debconf translations, and such obvious fixes, we
shouldn't waste our time).
There's a middle ground between the
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 29/05/15 06:41, Haïkel wrote: Here's the main script to rebuild a RPM package. https://github.com/openstack-packages/delorean/blob/master/scripts/build_rpm.sh The script basically uses rpmbuild to build packages, we could have a build_deb.sh that uses sbuild and add dockerfiles for the Debian/Ubuntu supported releases. I have a preliminary patch locally that adds support for building for both Debian and Ubuntu. I will be applying some polish next week and then working with the Delorean guys to get it landed. -- Steve Oh, in case you got covered in that Repulsion Gel, here's some advice the lab boys gave me: [paper rustling] DO NOT get covered in the Repulsion Gel. - Cave Johnson, CEO of Aperture Science __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Derek, Thanks for what you wrote. On 05/27/2015 11:26 PM, Derek Higgins wrote: 4. For deb packages you can create new repositories along side the rdorpm-* repositories My intention is to use deb-* as prefix, if Canonical team agrees. 5. Add deb support to delorean, I know of at least one person who has already explored this (steve cc'd), if delorean is too far off the path of what we want to achieve and there is a better tool then I'm open to change. I don't know delorean at all, but what should be kept in mind is that, for Debian and Ubuntu, we *must* use sbuild, which is what is used on the buildd networks. I also started working on openstack-pkg-tools to provide such sbuild based build env, so I'm not sure if we need to switch to Delorean. Could you point me to some documentation about it, so I can see by myself what Delorean is about? Cheers, Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Hi Thomas, Delorean is a tool to build rpm packages from master branches (maybe any branch?) of OpenStack projects. Check out here: https://www.rdoproject.org/packaging/rdo-packaging.html#master-pkg-guide Regards, On Thu, 28 May 2015 10:40, Thomas Goirand wrote: Derek, Thanks for what you wrote. On 05/27/2015 11:26 PM, Derek Higgins wrote: 4. For deb packages you can create new repositories along side the rdorpm-* repositories My intention is to use deb-* as prefix, if Canonical team agrees. 5. Add deb support to delorean, I know of at least one person who has already explored this (steve cc'd), if delorean is too far off the path of what we want to achieve and there is a better tool then I'm open to change. I don't know delorean at all, but what should be kept in mind is that, for Debian and Ubuntu, we *must* use sbuild, which is what is used on the buildd networks. I also started working on openstack-pkg-tools to provide such sbuild based build env, so I'm not sure if we need to switch to Delorean. Could you point me to some documentation about it, so I can see by myself what Delorean is about? Cheers, Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Jaume Devesa Software Engineer at Midokura __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 28/05/15 12:07, Jaume Devesa wrote: Hi Thomas, Delorean is a tool to build rpm packages from master branches (maybe any branch?) of OpenStack projects. Check out here: https://www.rdoproject.org/packaging/rdo-packaging.html#master-pkg-guide Following those instructions you'll notice that the rpm's are being built using rpmbuild inside a docker container, if expanding to add dep support this is where we could plug in sbuild. Regards, On Thu, 28 May 2015 10:40, Thomas Goirand wrote: Derek, Thanks for what you wrote. On 05/27/2015 11:26 PM, Derek Higgins wrote: 4. For deb packages you can create new repositories along side the rdorpm-* repositories My intention is to use deb-* as prefix, if Canonical team agrees. 5. Add deb support to delorean, I know of at least one person who has already explored this (steve cc'd), if delorean is too far off the path of what we want to achieve and there is a better tool then I'm open to change. I don't know delorean at all, but what should be kept in mind is that, for Debian and Ubuntu, we *must* use sbuild, which is what is used on the buildd networks. I also started working on openstack-pkg-tools to provide such sbuild based build env, so I'm not sure if we need to switch to Delorean. Could you point me to some documentation about it, so I can see by myself what Delorean is about? Cheers, Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 05/28/2015 01:07 PM, Jaume Devesa wrote: Hi Thomas, Delorean is a tool to build rpm packages from master branches (maybe any branch?) of OpenStack projects. It's now also used for stable/kilo. I suspect all supported branches starting from Kilo will be consumed by Delorean. Ihar -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJVZwRLAAoJEC5aWaUY1u57jEcH+gKPNa0RNfMmSUrJTI3fD+SK 7vHV1qRUww3VNzJbSF0Tdr8Zp9KsIIarXgZMyCPM4i/0Syy1Gr5PVCVnRX8TjK/V g5NwQ69KXAiEbSCrbUJVZx4fkC4SW3E9h/UPkkaCd2IYniTv7/KeoKRSX1lHHxrl eQSBxn4gyl2r3F7Ilq5uPb1fYSOXonlGwIFEFuyS/FjXppZ1SK0V+A3DIUavt8Tz 27S8HqC6oxwbac/iABgSYF1f0vp//W0NJSosgTV2PUrlycBLTX7o4uAe/0dEmc7K z+aTIaa3rrb44htzrGYIF4T3JHencASoPpDq/yRa7mhSX7EHjRENV7154pJJ75U= =IrKJ -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Great initiative, IMO. I favour going directly to openstack-, rather than
stackforge-, for the migration reason that you mention.
Original Message
From: Thomas Goirand
Sent: Wednesday, 27 May 2015 09:17
To: OpenStack Development Mailing List (not for usage questions)
Reply To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [packaging] Adding packaging as an OpenStack project
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi all,
tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the overall
QA for packaging *and* upstream.
- - The goal is *not* to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)
Longer version:
Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.
During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.
A number of topics have been raised, which needs to be shared.
First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.
So here it is. Suse people (and other distributions), you're welcome to
join the effort.
- - Why doing this
It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
that we'd be a way more effective if we worked better together, on a
collaborative fashion using a review process like on upstream Gerrit.
But also, we'd like to welcome anyone, and especially the operation
folks, to contribute and give feedback. Using Gerrit is the obvious way
to give everyone a say on what we're implementing.
As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.
This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.
We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.
Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already *before*
this is implemented).
- - What it will *not* be
===
We do not have the intention (yet?) to publish the resulting packages
built on upstream infra. Yes, we will share the same Git repositories,
and yes, the infra will need to keep a copy of all builds (for example,
because core packages will need oslo.db to build and run unit tests).
But we will still upload on each distributions on separate repositories.
So published packages by the infra isn't currently discussed. We could
get to this topic once everything is implemented, which may be nice
(because we'd have packages following trunk), though please, refrain to
engage in this topic right now: having the implementation done is more
important for the moment. Let's try to stay on tracks and be constructive.
- - Let's keep efficiency in mind
===
Over the last few years, I've been able to maintain all of OpenStack in
Debian with little to no external contribution. Let's hope that the
Gerrit workflow will not slow down too much the packaging work, even if
there's an unavoidable overhead. Hopefully, we can implement some
liberal ACL policies for the core reviewers so that the Gerrit workflow
don't slow down anyone too much. For example we may be able to create
new repositories very fast, and it may
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
Many thanks to Thomas and the other packagers for a great discussion at
the summit and this fast follow-up, explained well. Looking forward to
seeing what can be achieved!
On 27/05/15 16:14, Thomas Goirand wrote:
Hi all,
tl;dr:
- We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- The intention is to better share the workload, and improve the overall
QA for packaging *and* upstream.
- The goal is *not* to publish packages upstream
- There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- Finally, let's not discuss for too long and let's do it!!! :)
Longer version:
Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.
During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.
A number of topics have been raised, which needs to be shared.
First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.
So here it is. Suse people (and other distributions), you're welcome to
join the effort.
- Why doing this
It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
that we'd be a way more effective if we worked better together, on a
collaborative fashion using a review process like on upstream Gerrit.
But also, we'd like to welcome anyone, and especially the operation
folks, to contribute and give feedback. Using Gerrit is the obvious way
to give everyone a say on what we're implementing.
As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.
This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.
We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.
Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already *before*
this is implemented).
- What it will *not* be
===
We do not have the intention (yet?) to publish the resulting packages
built on upstream infra. Yes, we will share the same Git repositories,
and yes, the infra will need to keep a copy of all builds (for example,
because core packages will need oslo.db to build and run unit tests).
But we will still upload on each distributions on separate repositories.
So published packages by the infra isn't currently discussed. We could
get to this topic once everything is implemented, which may be nice
(because we'd have packages following trunk), though please, refrain to
engage in this topic right now: having the implementation done is more
important for the moment. Let's try to stay on tracks and be constructive.
- Let's keep efficiency in mind
===
Over the last few years, I've been able to maintain all of OpenStack in
Debian with little to no external contribution. Let's hope that the
Gerrit workflow will not slow down too much the packaging work, even if
there's an unavoidable overhead. Hopefully, we can implement some
liberal ACL policies for the core reviewers so that the Gerrit workflow
don't slow down anyone too much. For example we may be able to create
new repositories very fast, and it may be possible to self-approve some
of the most trivial patches (for things like typo in a package
description, adding new debconf translations, and such obvious fixes, we
shouldn't
Re: [openstack-dev] [packaging] Adding packaging as an OpenStack project
On 27/05/15 09:14, Thomas Goirand wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hi all,
tl;dr:
- - We'd like to push distribution packaging of OpenStack on upstream
gerrit with reviews.
- - The intention is to better share the workload, and improve the overall
QA for packaging *and* upstream.
- - The goal is *not* to publish packages upstream
- - There's an ongoing discussion about using stackforge or openstack.
This isn't, IMO, that important, what's important is to get started.
- - There's an ongoing discussion about using a distribution specific
namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
/stackforge-pkg-{deb,rpm} would be the most convenient because of a
number of technical reasons like the amount of Git repository.
- - Finally, let's not discuss for too long and let's do it!!! :)
Longer version:
Before I start: some stuff below is just my own opinion, others are just
given facts. I'm sure the reader is smart enough to guess which is what,
and we welcome anyone involved in the project to voice an opinion if
he/she differs.
During the Vancouver summit, operation, Canonical, Fedora and Debian
people gathered and collectively expressed the will to maintain
packaging artifacts within upstream OpenStack Gerrit infrastructure. We
haven't decided all the details of the implementation, but spent the
Friday morning together with members of the infra team (hi Paul!) trying
to figure out what and how.
A number of topics have been raised, which needs to be shared.
First, we've been told that such a topic deserved a message to the dev
list, in order to let groups who were not present at the summit. Yes,
there was a consensus among distributions that this should happen, but
still, it's always nice to let everyone know.
So here it is. Suse people (and other distributions), you're welcome to
join the effort.
- - Why doing this
It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
that we'd be a way more effective if we worked better together, on a
collaborative fashion using a review process like on upstream Gerrit.
But also, we'd like to welcome anyone, and especially the operation
folks, to contribute and give feedback. Using Gerrit is the obvious way
to give everyone a say on what we're implementing.
As OpenStack is welcoming every day more and more projects, it's making
even more sense to spread the workload.
This is becoming easier for Ubuntu guys as Launchpad now understand not
only BZR, but also Git.
We'd start by merging all of our packages that aren't core packages
(like all the non-OpenStack maintained dependencies, then the Oslo libs,
then the clients). Then we'll see how we can try merging core packages.
Another reason is that we believe working with the infra of OpenStack
upstream will improve the overall quality of the packages. We want to be
able to run a set of tests at build time, which we already do on each
distribution, but now we want this on every proposed patch. Later on,
when we have everything implemented and working, we may explore doing a
package based CI on every upstream patch (though, we're far from doing
this, so let's not discuss this right now please, this is a very long
term goal only, and we will have a huge improvement already *before*
this is implemented).
- - What it will *not* be
===
We do not have the intention (yet?) to publish the resulting packages
built on upstream infra. Yes, we will share the same Git repositories,
and yes, the infra will need to keep a copy of all builds (for example,
because core packages will need oslo.db to build and run unit tests).
But we will still upload on each distributions on separate repositories.
So published packages by the infra isn't currently discussed. We could
get to this topic once everything is implemented, which may be nice
(because we'd have packages following trunk), though please, refrain to
engage in this topic right now: having the implementation done is more
important for the moment. Let's try to stay on tracks and be constructive.
- - Let's keep efficiency in mind
===
Over the last few years, I've been able to maintain all of OpenStack in
Debian with little to no external contribution. Let's hope that the
Gerrit workflow will not slow down too much the packaging work, even if
there's an unavoidable overhead. Hopefully, we can implement some
liberal ACL policies for the core reviewers so that the Gerrit workflow
don't slow down anyone too much. For example we may be able to create
new repositories very fast, and it may be possible to self-approve some
of the most trivial patches (for things like typo in a package
description, adding new debconf translations, and such obvious fixes, we
shouldn't waste our time).
There's a middle ground between the current system (ie: only write
access ACLs for git.debian.org with no other check what so ever) and a
too restrictive fully protected gerrit
