On Sunday 11 November 2007, Martijn Faassen wrote: > What KGS solves is that it allows the ongoing development and testing of > an integrated Zope 3. That is, there's a Zope 3 'trunk' of versions > that keeps being updated as there are bugfix releases. I'm not sure what > happens as soon as someone wants to make a new feature release of any > package. Presumably they end up in KGS too.
Absolutely not! Like Linux distributions, there will be a KGS for every Zope 3 release. I have already requested a new directory called "zope-dev" where new feature releases can be tested. > After all, we won't have a > single Zope 3.4 and then a single Zope 3.5 for which we can create a new > KGS. Yes, we will. Why do you think the current KGS is called "zope3.4"? If you want to have a different working set, then you are free to create one, but don't expect much support from the community when things are not working as expected. > We intend to let packages move at different feature-release speeds, > and we can't have a KGS for each package. You do not need to have a single KGS for every package. But believing that we can just randomly make new feature releases that work with the rest of the world is naive at best. We have seen already what happens, if everyone uses their own set of versions and packages. A development KGS will be used to test new feature releases. > What KGS doesn't have is history. Yes, it does. Why do you think I manage the "controlled-packages.cfg" file in SVN? And in SVN, I do not create branches and tags without a reason. > When I release an application or > framework and I used KGS to make sure that all my versions were correct, > it will work on the day of release. As soon as enough bugfix (or > feature) releases make it to KGS, something will inevitably break. We've > seen innocuous changes breaking code a lot of times, so we can't pretend > that never happens. It *will* happen. I agree. Have you read the discussion we had yesterday on the zope-dev mailing list? We discussed the problem and possible solutions already. Here are a couple of choices we have to avoid the problem: 1. During development I would recommend to use the index of the latest stable release; or if you are brave, you can use the development KGS. (Of course, you can also use the versions block of a particular release, though you will miss out on bug fixes, which I think is less optimal.) 2. Once release/deployment time comes around, you lock the versions. There is a wide range of possibilities: (a) You download the "version.cfg" of the KGS at this time and maintain it in your deployment code. (b) You point to a particular release's "versions.cfg", for example "versions-3.4.0.cfg". (I will start producing those starting with the next Zope 3.4 release. Maybe I should create a versioned file for "controlled-packages.cfg" as well?!) (c) You use a particular SVN revision, download `zope.release` yourself and generate the "versions.cfg" file, which is trivial. I already create tags for releases there. I probably would prefer option (b). > This breaks a fundamental assumption for releases. When I release > something, I expect it to work tomorrow, next month, and next year. I agree. The KGS should be seen as a branch. Particular versions of "versions.cfg" and maybe "controlled-packages.cfg" should be considered releases. > With code, we know that history, and branches, and so on, are important. > We use Subversion. With KGS we only have an ongoing trunk. No, as I said before, the KGS specification, which is "controlled-packages.cfg", is maintained in SVN as well. > With Grok, we use an external versions list. We can use this to solve > the above problem. We basically take snapshots of what is in KGS. This > allows us to maintain some history, though it isn't ideal either, as > it's quite a bit of overhead. How is this overhead? > If I build an application or framework on top of Grok, I will need to > maintain yet another external list for the extra packages of this > application, fixing those versions. Why? I don't follow that? > We could probably even use the > extends feature of buildout to have this list point at Grok's list so we > have to repeat ourselves less should we want to build something on top > of *that* application or framework again. I don't understand what you are saying. However, I'll note that the KGS is also extendable. For example, Grok can maintain its own "controlled-packages.cfg" that extends a particular Zope 3 "controlled-packages.cfg". Extending also means that you have the choice of overwriting a particular version requirement. (I have implemented this after the discussion yesterday.) Having a "controlled-packages.cfg" does *not* mean you need an index. This file can be used to generate a "versions.cfg" file or just a `[versions]` section for buildout. > So, while annoying, that is somewhat manageable. Now imagine I want to > use a completely separate Python library with my Grok application. This > python library has dependencies itself again. This means I will need to > know about versions of those dependencies as well, and fix them into my > application's list. Yes. I see this as an advantage. Version specifications in `setup.py` usually contain ranges of allowed versions. What happens if one release in the range does not work? Then you make false promises. The only way to avoid this would be by specifying all allowed versions exactly, which makes no sense. > There are some fundamental problems with external lists or indexes: > > * we need to know about the dependency of dependencies, even if we never > use them directly. Information hiding is broken. This is a requirement not a problem statement. I don't understand "Information hiding is broken." I cannot see how "information hiding" could be a good thing here. > * a single list will never do it. We intend to have many different > applications that may depend on different versions of packages. Grok may > need a newer zope.publication than your application does. A Grok > extension may need an even newer version than Grok does. We'll be baking > endless amounts of lists this way. I never claimed that we will have just one list. Your assumption is that there will be an endless amount of working sets and that they are easy to find. While theoretically possible, this will not happen. I think the events of the past two months have shown that the opposite happens and it is actually hard to find working sets, once development digresses too much. It is pretty easy to maintain a list for a particular application, especially with having already base lists, like zope3.4 and grok, being available. > If this information is inside the packages itself, the history will be > automatically maintained with Subversion and existing releases. History > therefore works: if I install Grok 0.11, I would get all dependencies of > Grok 0.11 automatically without having to worry about external indexes. This is the overly simplistic world view that we had about two months ago. Because we are all developing on top of different stacks and people expect to pick and choose, having a common foundation is the only possible way. Let me give you a dire scenario. Let's say you have a package A-1.0.0. You also have a package B-1.0.0 that depends on A. You suggest fixing versions, so you would write in the dep list of B: 'A => 1.0.0'. You now release a new feature version of A, A-1.1.0, that is incompatible with B-1.0.0. So package B-1.0.0 will be broken until you release B-1.0.1 that states 'A >= 1.0.0 and A <= 1.0.99'. This problem here is that you have to re-release B only for the sake of changing the version requirement. This is not so bad if you have one known package to do this with. But in Zope's case we often 20, 30, 60, or even 80 packages that now have to be re-released. All this work for one update. I can tell you that releasing this many packages is a very tedious job and very error-prone. I have just spent 3-4 man-weeks releasing about 120 packages, some multiple times. But this is not your biggest problem. You cannot even assume that you can know the full list of packages that need updating, because they are not public or you are not aware of them. And the worst about it all is that everyone will be blocked until the new releases are all out _, unless they do some internal version nailing for their application. Which brings us back to a KGS. Now, you could have simply skipped specifying versions in 'setup.py' and get the same effect. Specifying versions in the release makes you believe you are safe. But what you are really doing is to impose one underspecified working set onto everyone. But this working set is not universal; in fact it is very small, since it does not consider a larger dependency network. The Zope 3 KGS attempts to be a universal set for Zope 3, which other projects can built upon. ..  This did actually happened during the FoilageSprint. Remember all the outrage? > Information hiding works: if I use foo 1.3 and foo 1.3 knows it needs > bar 1.7, it'll simply get that and I don't have to know about it. I > don't even need to worry about the *existence* of bar. With KGS, you do not need to know about it either. One assumption you make is that bar 1.7 can be found in the default index (PyPI) or in any of the dependency links locations. In the KGS we simply change this to use a different index with the promise that only versions of bar are available that actually work, which might be bar 1.6, bar 1.7, or bar 1.8. > People have been saying that since Linux distributions use external > indexes, we should too, as we are dealing with the same problem as Linux > distributions. It is true that our problem is very much like a Linux distribution, at least my mind. However, Jim and I did not choose the index approach because of Linux. We simply thought about the easiest way to create a known-working-set that would require the least amount of software to be written. So Jim had the idea of simply limiting the amount of available versions of a package and noted that it would be a fairly simple (now about 50 lines of code) to extend his ppix tool to do that. Actually, the KGS is really only the "controlled-packages.cfg" file and the `http://download.zope.org/zope3.4` index is just one way to use the KGS. Other usages include the "versions" section generator, the test buildout generator, and the Zope 3 tree updater. > While the problem is similar, I think the nature of > development makes our problems, and therefore our solutions, quite > different from the way distributions do it. > > How are we different? That's a good question. I think you forgot the most important one. Manpower! In a Linux distribution, there are many contributors to just the release process. In the Python/Zope world, the developer is the release person too. Thus Linux distributions have the luxury that they can create a new release for every package for every Linux distribution release. One promise Jim and others made to me about eggs was that once we had a stable set (which I worked on for the current KGS), releasing would become much simpler, because many packages would not need new releases for very long stretches at a time. With your suggestion, packages will need to be released all the time. > We have many, many different small distributions (package + > dependencies) that can be combined. We have such a small distribution > for each application. We have such a small distribution for each > extension. Not just that. We have such a small distribution for each > *release* of an application. We have such a small distribution for each > *release* of an extension. I think your proposal makes this a problem, whereby the KGS provides a solution; of course, in combination with the versions and find-links options. > I therefore still believe that version dependency information should > move out of external indexes and into packages. - 1 googol Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )