Re: [Openstack] Single global dependency list
Joshua Harlow wrote: Ack, please don’t keep on adding on to the copy around stuff scheme. Plese :-) You might have misread what Monty proposes. openstack-requires would actually duplicate the openstack-common copy-around mechanism (update.py) in an even more permanent way (openstack-common will become a library at the end of time, whereas openstack-requires won't). -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Single global dependency list
On Jul 2, 2012, at 6:40 PM, Monty Taylor mord...@inaugust.com wrote: Hey all! One of the tasks from the last ODS was to implement a single global dependency list. Turns out the more you think about it, the more important it is... because of the way we use devstack as part of the gate, we actually _currently_ have a de facto global dependency list, it's just not declared anywhere. (oops) Anyway - the original thought was to put the depends in openstack-common. We'd use update.py to copy the depends in to the project, so that projects could align on their own timeframe. Additionally, we'd make the copy only copy in the versions from openstack-common for package that were already listed in the target project, so that we wouldn't add django to python-swiftclient, for instance. The mechanics of that all work and are ready - but then bcwaldon pointed out that it didn't make a ton of sense for them to go in openstack-common, since that has its own lifecycle and is a place for common code to go - not just a catch all place. To that end, I took the code we had written for the update logic and put it, along with the depends lists, into its own repo. I think we're ready to start actually moving forward with it - but we've run up against the hardest problem we every have: naming openstack-depends already got vetoed on IRC because it makes people think of adult diapers. I'm proposing openstack-requires, since the files we're talking about are actually python requirements files. Any objections? +0 on the name As an alternative, how about combining the requirements file with the other packaging related stuff from openstack-common and calling the result openstack-packaging? Doug ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Single global dependency list
On 07/03/2012 08:43 AM, Doug Hellmann wrote: On Jul 2, 2012, at 6:40 PM, Monty Taylor mord...@inaugust.com wrote: Hey all! One of the tasks from the last ODS was to implement a single global dependency list. Turns out the more you think about it, the more important it is... because of the way we use devstack as part of the gate, we actually _currently_ have a de facto global dependency list, it's just not declared anywhere. (oops) Anyway - the original thought was to put the depends in openstack-common. We'd use update.py to copy the depends in to the project, so that projects could align on their own timeframe. Additionally, we'd make the copy only copy in the versions from openstack-common for package that were already listed in the target project, so that we wouldn't add django to python-swiftclient, for instance. The mechanics of that all work and are ready - but then bcwaldon pointed out that it didn't make a ton of sense for them to go in openstack-common, since that has its own lifecycle and is a place for common code to go - not just a catch all place. To that end, I took the code we had written for the update logic and put it, along with the depends lists, into its own repo. I think we're ready to start actually moving forward with it - but we've run up against the hardest problem we every have: naming openstack-depends already got vetoed on IRC because it makes people think of adult diapers. I'm proposing openstack-requires, since the files we're talking about are actually python requirements files. Any objections? +0 on the name As an alternative, how about combining the requirements file with the other packaging related stuff from openstack-common and calling the result openstack-packaging? Interesting. Which other packaging stuff? Do you mean the stuff in openstack/common/setup.py? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Single global dependency list
I have to agree with others that copying files around is not ideal, and I can see the maintenance of this getting more involved as Nova becomes more coupled with common. Additionally, we'd make the copy only copy in the versions from openstack-common for package that were already listed in the target project, so that we wouldn't add django to python-swiftclient, for instance. This seems to be a reasonable argument against using git submodules, but I'm afraid we might be losing more than we're gaining here. Just because python-swiftclient depends on openstack-common, and django-using code exists there, doesn't mean that django needs to be installed for python-swiftclient. We might do better to use git submodules and solve the dependency problem, than continuing down this copy-everything path. Alternatively, speed up the movement from incubation to library. Regards, Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Single global dependency list
On 07/03/2012 10:09 AM, Eric Windisch wrote: I have to agree with others that copying files around is not ideal, and I can see the maintenance of this getting more involved as Nova becomes more coupled with common. Additionally, we'd make the copy only copy in the versions from openstack-common for package that were already listed in the target project, so that we wouldn't add django to python-swiftclient, for instance. This seems to be a reasonable argument against using git submodules, but I'm afraid we might be losing more than we're gaining here. Just because python-swiftclient depends on openstack-common, and django-using code exists there, doesn't mean that django needs to be installed for python-swiftclient. We might do better to use git submodules and solve the dependency problem, than continuing down this copy-everything path. We're explicitly NOT doing a copy-everything path. That's the whole point. We're only copying the needed depends from the master list. git submodules actually make the problem worse, not better. Alternatively, speed up the movement from incubation to library. Yeah - that's kind of the reason that bcwaldon was saying this shouldn't be in openstack-common. openstack-common wants to be a library, and then we're back at not having an appropriate place for the master list. Monty ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Single global dependency list
So, I understand the rationales, and I think of those three options the one chosen is the most reasonable. I'm gonna just come out and say I hate this whole idea, but let's set this aside for a minute 'cuz I have a genuine question: What will the process be for merging changes to this requirements list? Having yet another roadblock to getting your contribution merged is a huge developer disincentive. We're really making it exceptionally hard for new contributors, and frustrating even for the old hands. So, with the goal of making the coordinated management of the projects possible, what can we do to respect developers? - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Monty Taylor Sent: Tuesday, July 03, 2012 7:54 AM To: Eric Windisch Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Single global dependency list On 07/03/2012 10:09 AM, Eric Windisch wrote: I have to agree with others that copying files around is not ideal, and I can see the maintenance of this getting more involved as Nova becomes more coupled with common. Additionally, we'd make the copy only copy in the versions from openstack-common for package that were already listed in the target project, so that we wouldn't add django to python-swiftclient, for instance. This seems to be a reasonable argument against using git submodules, but I'm afraid we might be losing more than we're gaining here. Just because python-swiftclient depends on openstack-common, and django-using code exists there, doesn't mean that django needs to be installed for python-swiftclient. We might do better to use git submodules and solve the dependency problem, than continuing down this copy-everything path. We're explicitly NOT doing a copy-everything path. That's the whole point. We're only copying the needed depends from the master list. git submodules actually make the problem worse, not better. Alternatively, speed up the movement from incubation to library. Yeah - that's kind of the reason that bcwaldon was saying this shouldn't be in openstack-common. openstack-common wants to be a library, and then we're back at not having an appropriate place for the master list. Monty ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Single global dependency list
It's a good and valid question and I don't really know. In this case, I'm merely the pack-horse who was told global synchronized dependencies lists! (not that I'm not the evil person cooking up schemes) That said - most patches from new contributors don't actually come with new library dependencies. If they are adding a new depend, I think it's reasonable to expect it to be slightly harder to get that landed. I do think that we need an answer to who approves changes to this list. Getting stuff merged to openstack-common is often hard because it's a smaller list of people who work on it. I'd hate to see this be only PTLs. However, things like let's upgrade webob seem to _actually_ need more eyes than it seems like at the time. meh. On 07/03/2012 03:12 PM, Gabriel Hurley wrote: So, I understand the rationales, and I think of those three options the one chosen is the most reasonable. I'm gonna just come out and say I hate this whole idea, but let's set this aside for a minute 'cuz I have a genuine question: What will the process be for merging changes to this requirements list? Having yet another roadblock to getting your contribution merged is a huge developer disincentive. We're really making it exceptionally hard for new contributors, and frustrating even for the old hands. So, with the goal of making the coordinated management of the projects possible, what can we do to respect developers? - Gabriel -Original Message- From: openstack-bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Monty Taylor Sent: Tuesday, July 03, 2012 7:54 AM To: Eric Windisch Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Single global dependency list On 07/03/2012 10:09 AM, Eric Windisch wrote: I have to agree with others that copying files around is not ideal, and I can see the maintenance of this getting more involved as Nova becomes more coupled with common. Additionally, we'd make the copy only copy in the versions from openstack-common for package that were already listed in the target project, so that we wouldn't add django to python-swiftclient, for instance. This seems to be a reasonable argument against using git submodules, but I'm afraid we might be losing more than we're gaining here. Just because python-swiftclient depends on openstack-common, and django-using code exists there, doesn't mean that django needs to be installed for python-swiftclient. We might do better to use git submodules and solve the dependency problem, than continuing down this copy-everything path. We're explicitly NOT doing a copy-everything path. That's the whole point.. We're only copying the needed depends from the master list. git submodules actually make the problem worse, not better. Alternatively, speed up the movement from incubation to library. Yeah - that's kind of the reason that bcwaldon was saying this shouldn't be in openstack-common. openstack-common wants to be a library, and then we're back at not having an appropriate place for the master list. Monty ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Single global dependency list
Agreed on all points, and I know you're not evil, Monty. ;-) (mostly) You're totally right that this particular case won't stymie new contributors, but as we've seen for changes to common--and sometimes even to the client libraries or devstack--reviewers are in short supply and getting the change you need in one of the gate projects merged can often add days of impedance to otherwise fruitful work. It's bitten me plenty of times. So the need for balance is critical. Being able to vet the impact of a change on every project consuming it is difficult for either automated systems or human reviewers, so we do our best. Perhaps the simplest answer for now is devising a reasonable set of automated gate tests for this os-requires module that humans can trust, and working to expand the circle of reviewers on these centralized projects that have the power to block everyone yet are so easy to ignore... All the best, - Gabriel -Original Message- From: Monty Taylor [mailto:mord...@inaugust.com] Sent: Tuesday, July 03, 2012 12:49 PM To: Gabriel Hurley Cc: Eric Windisch; openstack@lists.launchpad.net Subject: Re: [Openstack] Single global dependency list It's a good and valid question and I don't really know. In this case, I'm merely the pack-horse who was told global synchronized dependencies lists! (not that I'm not the evil person cooking up schemes) That said - most patches from new contributors don't actually come with new library dependencies. If they are adding a new depend, I think it's reasonable to expect it to be slightly harder to get that landed. I do think that we need an answer to who approves changes to this list. Getting stuff merged to openstack-common is often hard because it's a smaller list of people who work on it. I'd hate to see this be only PTLs. However, things like let's upgrade webob seem to _actually_ need more eyes than it seems like at the time. meh. On 07/03/2012 03:12 PM, Gabriel Hurley wrote: So, I understand the rationales, and I think of those three options the one chosen is the most reasonable. I'm gonna just come out and say I hate this whole idea, but let's set this aside for a minute 'cuz I have a genuine question: What will the process be for merging changes to this requirements list? Having yet another roadblock to getting your contribution merged is a huge developer disincentive. We're really making it exceptionally hard for new contributors, and frustrating even for the old hands. So, with the goal of making the coordinated management of the projects possible, what can we do to respect developers? - Gabriel -Original Message- From: openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Monty Taylor Sent: Tuesday, July 03, 2012 7:54 AM To: Eric Windisch Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Single global dependency list On 07/03/2012 10:09 AM, Eric Windisch wrote: I have to agree with others that copying files around is not ideal, and I can see the maintenance of this getting more involved as Nova becomes more coupled with common. Additionally, we'd make the copy only copy in the versions from openstack-common for package that were already listed in the target project, so that we wouldn't add django to python-swiftclient, for instance. This seems to be a reasonable argument against using git submodules, but I'm afraid we might be losing more than we're gaining here. Just because python-swiftclient depends on openstack-common, and django-using code exists there, doesn't mean that django needs to be installed for python-swiftclient. We might do better to use git submodules and solve the dependency problem, than continuing down this copy-everything path. We're explicitly NOT doing a copy-everything path. That's the whole point.. We're only copying the needed depends from the master list. git submodules actually make the problem worse, not better. Alternatively, speed up the movement from incubation to library. Yeah - that's kind of the reason that bcwaldon was saying this shouldn't be in openstack-common. openstack-common wants to be a library, and then we're back at not having an appropriate place for the master list. Monty ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Single global dependency list
On 07/03/2012 04:23 PM, Gabriel Hurley wrote: Agreed on all points, and I know you're not evil, Monty. ;-) (mostly) Dammit. I'm going to HAVE to try harder... see my other post about Bulgarian bars with freezers... You're totally right that this particular case won't stymie new contributors, but as we've seen for changes to common--and sometimes even to the client libraries or devstack--reviewers are in short supply and getting the change you need in one of the gate projects merged can often add days of impedance to otherwise fruitful work. It's bitten me plenty of times. Yes. It's super hard and frustrating- especially as someone who rarely makes patches to the projects themselves UNLESS it's something happening through openstack-common ... or something that just broke between devstack and our infrastructure. So the need for balance is critical. Being able to vet the impact of a change on every project consuming it is difficult for either automated systems or human reviewers, so we do our best. Perhaps the simplest answer for now is devising a reasonable set of automated gate tests for this os-requires module that humans can trust, and working to expand the circle of reviewers on these centralized projects that have the power to block everyone yet are so easy to ignore... TOTALLY agree - especially on expanding the circle of reviewers. All the best, - Gabriel -Original Message- From: Monty Taylor [mailto:mord...@inaugust.com] Sent: Tuesday, July 03, 2012 12:49 PM To: Gabriel Hurley Cc: Eric Windisch; openstack@lists.launchpad.net Subject: Re: [Openstack] Single global dependency list It's a good and valid question and I don't really know. In this case, I'm merely the pack-horse who was told global synchronized dependencies lists! (not that I'm not the evil person cooking up schemes) That said - most patches from new contributors don't actually come with new library dependencies. If they are adding a new depend, I think it's reasonable to expect it to be slightly harder to get that landed. I do think that we need an answer to who approves changes to this list. Getting stuff merged to openstack-common is often hard because it's a smaller list of people who work on it. I'd hate to see this be only PTLs. However, things like let's upgrade webob seem to _actually_ need more eyes than it seems like at the time. meh. On 07/03/2012 03:12 PM, Gabriel Hurley wrote: So, I understand the rationales, and I think of those three options the one chosen is the most reasonable. I'm gonna just come out and say I hate this whole idea, but let's set this aside for a minute 'cuz I have a genuine question: What will the process be for merging changes to this requirements list? Having yet another roadblock to getting your contribution merged is a huge developer disincentive. We're really making it exceptionally hard for new contributors, and frustrating even for the old hands. So, with the goal of making the coordinated management of the projects possible, what can we do to respect developers? - Gabriel -Original Message- From: openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net [mailto:openstack- bounces+gabriel.hurley=nebula@lists.launchpad.net] On Behalf Of Monty Taylor Sent: Tuesday, July 03, 2012 7:54 AM To: Eric Windisch Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Single global dependency list On 07/03/2012 10:09 AM, Eric Windisch wrote: I have to agree with others that copying files around is not ideal, and I can see the maintenance of this getting more involved as Nova becomes more coupled with common. Additionally, we'd make the copy only copy in the versions from openstack-common for package that were already listed in the target project, so that we wouldn't add django to python-swiftclient, for instance. This seems to be a reasonable argument against using git submodules, but I'm afraid we might be losing more than we're gaining here. Just because python-swiftclient depends on openstack-common, and django-using code exists there, doesn't mean that django needs to be installed for python-swiftclient. We might do better to use git submodules and solve the dependency problem, than continuing down this copy-everything path. We're explicitly NOT doing a copy-everything path. That's the whole point.. We're only copying the needed depends from the master list. git submodules actually make the problem worse, not better. Alternatively, speed up the movement from incubation to library. Yeah - that's kind of the reason that bcwaldon was saying this shouldn't be in openstack-common. openstack-common wants to be a library, and then we're back at not having an appropriate place for the master list. Monty ___ Mailing list: https://launchpad.net/~openstack Post to : openstack
