On Wed, Jul 9, 2014 at 7:53 AM, Daniel P. Berrange <berra...@redhat.com> wrote: > On Wed, Jul 09, 2014 at 07:39:39AM -0400, Sean Dague wrote: >> On 07/03/2014 01:52 PM, Doug Hellmann wrote: >> >> oslo code, by definition, is going to be used a lot, inside of tight >> >> loops. Which means extremely verbose. Actually >> >> nova.openstack.common.policy currently represents ~40% of all bytes in >> >> nova-api logs (15 MB uncompressed content during a Tempest run). >> >> >> >> Both these are bugs that should be fixed, however, I'd like to figure >> >> out how we come up with more systematic approach to keep the quality of >> >> these things up, especially as an oslo fix is expensive in terms of >> >> labor, as we have to fix in oslo first, then sync to projects. And as >> >> more syncs have caused issues like this an oslo sync is not longer a >> >> fast approve as it used to be. >> > >> > Do you have some automation in mind, or just focusing more on logging >> > in reviews? >> >> I don't know. I'm just concerned that some really bad bugs are slipping >> through oslo incubator, which is then copy / pasted into projects before >> they get noticed (because there is no forward testing), then we have to >> run around and replicate fixes everywhere. >> >> Stuff like this is generating a ton of extra work on everyone that's not >> oslo. > > This is yet another example of the many problems inherant in the cut+paste > approach used by olso incubator. I'd personally like to see olso-incubator > killed and everything there moved into its own module with whatever API > they have now. If we later find we need to break the APIs of some of the > split out modules, so be it. IMHO it would be less pain overall than we > have with the cut+paste approach. We'd be able to have fully integrated > CI testing running against the olso changes as they go on, rather than > only after copy+paste, whic would ease detection/avoid problems like you're > highlighting here.
The problem is not that we are copying code from the incubator. The problem is that too many of us are copying code into too many projects before it is mature. The lifecycle for incubated code is: 1. Someone in a project writes a module that works in that application. 2. Someone in another project wants something similar. 3. The code is moved to the incubator, and then copied into both apps. 4. The code is changed to decouple it from either application (removing access to global state, naming configuration options more generically and putting them in groups, adding configuration hooks so we don't have nova calls in glance, etc.). 5. The API for the code is minimized so that implementation details aren't exposed. 6. The code is turned into a proper library. A lot of contributions to Oslo have stopped at step 3, with people moving on to other work. Steps 4 & 5 cannot be done in a vacuum. They have to happen in the context of putting the code back into the application where those changes can be tested in place. As soon as a library is released, that API is effectively frozen for a long time, because if any project is using a piece of the API we can't remove it. This is why step 5 is so important. SemVer has been proposed as a way to make backwards-incompatible changes. SemVer lets you declare when those changes have happened, but it doesn't meet our requirements for installing all of OpenStack on a single server and handling rolling updates. Our commitment to supporting single-node installations means we need to have all of OpenStack installed with a single version of each library, from system packages, on one box. Managing rolling updates with different versions of two services are installed on the same server means we need to replace one of those libraries while keeping both services working. Those are seriously constraining compatibility requirements, and that is why the incubator exists and won't just go away. Think of using the incubator as "static linking" to a library. You get the benefit of code sharing, along with the benefit that if the API of that code changes you can decide when to take the changes. When the module's API is stable, it can be moved to a library and shared more dynamically. The Oslo team has worked through the graduation process with some of the bigger libraries (config and messaging), and we are doing so with the database code and some of the smaller modules this cycle. Along the way we're learning just how many assumptions these modules make about the context in which they are used (lots of global state), how many edge cases are not documented clearly (nova cells relies on locking semantics of a thread locking library provided by *file* locks on shared filesystems), where we have deployer-specific code (the incubated database code uses an eventlet threadpool implementation only available in a patched and unreleased version of eventlet), and how many places we have unit tests for application code mocking out internal bits of incubated modules (yes). All of those challenges can be overcome, but it's not as simple as packaging up what we have and calling it a library. The result of doing that wouldn't be usable by anyone. Doug > > Regards, > Daniel > -- > |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| > |: http://libvirt.org -o- http://virt-manager.org :| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
