On 07/02/2012 08:43 PM, Joshua Harlow wrote: > Ack, please don’t keep on adding on to the copy around stuff scheme. > Pleaaaase :-)
You know - I was a huge opponent to the copy around stuff scheme when it started. It raised all of my hackles and I got all upset about things not being done the "right" way. Turns out that we have a complex project with some interesting requirements, and sometimes having an automated way to inject filtered sets of code from a common source isn't as terrible a thing as you might think (especially considering how half-baked ALL of pythons library and dependency management stuff is. oy - I'm amazed I have hair left...) > Is the devstack dependency list complete, when I created the anvil one, > I found more than one hole... Hi! This has nothing to do with the devstack dependency list. This list is two lists, actually - one is the the superset of all of the pip-requires files in all of the projects, the other is the superset of all of the test-requires files in all of the projects. > Also the devstack one is in a micro-format, maybe we can move to say, YAML? The purpose of this is to drive pip installations of packages for virtualenvs. That already has a required format that we have to use. (again, this has nothing to do with devstack) > How about hosting these requires on some website (with versions there)? The apprpriate contents still need to exist in each projects' pip-requires and test-requires files for unit testing, so you'll still have to copy code into the projects. > Github already provides this for the anvil dependency list, maybe that > (or something) similar is sufficent? It's not even close. It's possible that I was not clear on the purpose of what this is trying to accomplish. The underlying problem is that each project declares a set of python-level dependencies. This is important so that the base unit of release, the source tarball, is itself both complete, and also testable, since we have to be able to have python install the necessary depends to run unittests. I know there are people who would prefer we did that from OS packages, but that cat is out of the bag and _way_ out of scope for this particular discussion. Suffice it to say, each project needs a list of depends that it can hand to pip (or in the case of the client libs, additionally express in setup.py so that when the client lib is installed via pip as a dependency of something else, all of its depends come as well) So thing number one is that there will be a python requirements.txt format file in each project. The problem at the point is keeping them in sync so that we don't have version drift, which is a thing we decided at ODS that we wanted to do. When I mentioned devstack in my earlier email, what I was referring to is the fact that, since devstack installs each of the projects in sequence, although we do not at the moment have a de jure global dependency list, in practice we run all of our integration testing on installs on a single machine - which means there _is_ a single set of packages and versions that everything is required to run from - we just don't declare it. Back to the specific implementation. There are three main ways we've come up with to solve having this list. The first had to do with git submodules. We discarded this because most people are still against the use of submodules in the project, and also because it makes testing coordinated changes quite difficult here. (if you want to increase the version of webob that the project uses, do you need to land that change in the depend list and support it in multiple projects that are consuming that list) The other problem is that if we just made the tools dir a submodule (or something similar) there would be no way to exclude certain depends from certain projects. The total list is quite large, and for many of our projects (swift and all of the client libs) it would introduce a massive additional unnecessary installation burden for unit testing. The second and third both involve copying, although the mechanism could be different. In one version, we would do as you suggest above and post the global depends file up on a website somewhere. Then, at runtime, the project could do a wget of the file, perform the merge into the local requirements list, and then go on about its business. It means that the merge code would run hundreds of time more, the projects would lose the ability to control when they sync up with the global depends, and it would make it much more brittle to use any of our projects. So we thought that since we've already gone down the hell-hole of copying code from openstack-common, copying requirements file contents at the same time won't kill us. Is that more helpful? Thanks! Monty > On 7/2/12 3:40 PM, "Monty Taylor" <mord...@inaugust.com > <mord...@inaugust..com>> wrote: > > Hey all! > > One of the tasks from the last ODS was to implement a single global > dependency list. Turns out the more you think about it, the more > important it is... because of the way we use devstack as part of the > gate, we actually _currently_ have a de facto global dependency list, > it's just not declared anywhere. (oops) > > Anyway - the original thought was to put the depends in > openstack-common. We'd use update.py to copy the depends in to the > project, so that projects could align on their own timeframe. > Additionally, we'd make the copy only copy in the versions from > openstack-common for package that were already listed in the target > project, so that we wouldn't add django to python-swiftclient, for > instance.. > > The mechanics of that all work and are ready - but then bcwaldon pointed > out that it didn't make a ton of sense for them to go in > openstack-common, since that has its own lifecycle and is a place for > common code to go - not just a catch all place. > > To that end, I took the code we had written for the update logic and put > it, along with the depends lists, into its own repo. I think we're ready > to start actually moving forward with it - but we've run up against the > hardest problem we every have: > > naming > > openstack-depends already got vetoed on IRC because it makes people > think of adult diapers. I'm proposing openstack-requires, since the > files we're talking about are actually python requirements files. > > Any objections? > > We've also been discussing an idea that we could write some gating tests > that are only run against milestone-proposed branches that would verify > that the requires files in a given project match the versions in > openstack-requires - that way there is some lee-way throughout the > cycle, but the expectation is that by release time the requires files > will be brought in line with the rest of the project. (it's an option if > people find that useful - certainly not required) > > Finally, as an added bonus to this approach, once we have the list and > we're even mostly aligned on it, since a library version would need to > be in openstack-requires before it hits a project, we can use it as the > main basis for our pypi mirror and turn off the upstream pypi source > from our jenkins slaves. This would remove the last of the ephemeral > build errors that are due to upstream network timeouts. I'm sure > everyone will be pleased about that. > > Anyway - I think general buy in on at _least_ the name > openstack-requires will let us move forward with the next step. After > that point, it'll all be normal code reviews. > > Monty > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp