Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
Hey, [eggs in debian] Okay, I can see this as an additional, more stably maintained resource of eggs than the cheeseshop. That might indeed be helpful. Now, how would you use the Grok gated community with the sqlalchemy gated community if they had common dependencies, and those dependencies were at different versions in the two communities? I don't know. :) I don't know how a gated community is supposed to work. Of course, you could make one _very_ large gated community and freeze it as a release every now and then. Er, basically the same as what distros are doing, hopefully making their life much easier;) One of the issues is that it doesn't solve my problem if you just have a big one. You'd need a gated community per *version* of Grok in order to make sure no newer eggs sneak into the project after release. Regards, Martijn ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
On Thu, Sep 27, 2007 at 11:11:06PM +0200, Martijn Faassen wrote: Hey, On 9/27/07, Brian Sutherland [EMAIL PROTECTED] wrote: [snip] There is one I thought of, but it's a bit backwards. Essentially, Debian has a repository of mostly unmodified original egg tarballs. And, they've already done the hard work of maintaining sane dependencies. So, why not simply re-name the .orig.tar.gz in a Debian release repository to their original names and you have a working set corresponding to that release. If you add your selected personal working set, then you have a basis for working without bleeding. Does it contain the Zope 3.4 eggs? No. But then my point wasn't that you would use it for Zope3 eggs, but for everything else. There's a lot of eggs, even in etch. And in lenny there will be even more. There are few pre-release/alpha/beta versions of eggs. In fact, a lot of maintainers prefer to release older/stabler versions than the latest release. And the only changes that happen to releases are security updates. Think of it as a stable base upon which to build. Will it have Zope 3.5 eggs when we need them? As above, not the point. I think if we're going to manage our own gated community, it'll be something we need to do ourselves. I just see the need for 1 per release of piece of software (grok 0.10, grok 0.11, etc), and only a vague idea how frameworks would work (I want to use the Foo gated community which also uses the Grok gated community) so I must be missing something. Now, how would you use the Grok gated community with the sqlalchemy gated community if they had common dependencies, and those dependencies were at different versions in the two communities? Of course, you could make one _very_ large gated community and freeze it as a release every now and then. Er, basically the same as what distros are doing, hopefully making their life much easier;) Regards, Martijn ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/jinty%40web.de -- Brian Sutherland ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
On Thu, Sep 27, 2007 at 10:09:23AM -0400, Jim Fulton wrote: On Sep 26, 2007, at 8:22 PM, Tres Seaver wrote: ... I think that replacing 'index_url' with a gated community of packages is the only path to sanity here: the contract of the Cheeseshop (share new releases of all packages with everyone ASAP) is incompatible with our goals (ensure that users can install a given package and its dependencies, and have them work). Regretfully, I agree. People here at ZC are already moving in this direction internally. It works for Linux distros: they periodically take packages from upstream (which would be Cheeseshop in our case) into their unstable distribution, fix incompatibilities there, then freeze and release a stable package set downloadable from a fixed URL. End users can use a stable version if they want things to Just Work, or they can install one or two packages from unstable if they really need a newer version of something, and are prepared to fix any incompatibilities themselves. Marius Gedminas -- To express oneself In seventeen syllables Is very diffic -- John Cooper Clark. signature.asc Description: Digital signature ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
Martijn Faassen wrote at 2007-9-26 22:13 +0200: ... I am the one who wants to have the final say in what versions of packages. I want to use. A linux distributor needs to have one working set of packages, instead. He may have one set of packages -- but he knows that not all of them work together. Moreover the set changes over time -- because new versions are released -- and finally the downloader is the one who has control over what he really installs on its local machine. Maybe, that's the equivalent to I am the one with the final say. ... We have a situation where we have developers, not maintainers, uploading new versions of packages. There will be no integrated testing done for all software built on all packages in the cheeseshop. Again, I can see similarities, but I don't believe linux distributions have *exactly* our problems solved. Our buildouts are used as development environments, not only deployment environments. Yes, we have less control over what is released on PyPI than a Linux distributor. But, you have control over what components your project depends on -- and you can select components based on underlying release care. Okay, you can also fix the dependency and thus skip careless releases... Sticking to stable versions helps, until a new stable version is released. Then all the old stuff suddenly starts using the *new* stable version, and probably break. You must have far worse experiences than I have. My components usually work across many releases with only very rare need to intervene (Five twice broke one of my products; I think that almost was it). -- Dieter ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
On Wed, Sep 26, 2007 at 08:22:48PM -0400, Tres Seaver wrote: Anybody running against the Cheeseshop today is *more* on the bleeding edge than a sysadmin whose production boxes are running 'sid': Debian has cultural constraits, even for that distro, which are vastly more restricted than the Wild West which is PyPI. The only solution I can see is to create filtered subsets / mirrors of PyPI. There is one I thought of, but it's a bit backwards. Essentially, Debian has a repository of mostly unmodified original egg tarballs. And, they've already done the hard work of maintaining sane dependencies. So, why not simply re-name the .orig.tar.gz in a Debian release repository to their original names and you have a working set corresponding to that release. If you add your selected personal working set, then you have a basis for working without bleeding. -- Brian Sutherland ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
Hey, We have a situation where we have developers, not maintainers, uploading new versions of packages. There will be no integrated testing done for all software built on all packages in the cheeseshop. Again, I can see similarities, but I don't believe linux distributions have *exactly* our problems solved. Our buildouts are used as development environments, not only deployment environments. Yes, we have less control over what is released on PyPI than a Linux distributor. But, you have control over what components your project depends on -- and you can select components based on underlying release care. Okay, you can also fix the dependency and thus skip careless releases... Exactly. I guess what I mean to say is that we're like mini distribution makers, we're not like users of distributions. Well, we're sort of both, as PyPI is a extremely messy distribution of sorts. We also typically have more than one mini distribution we maintain, while a Linux distro typically maintains only a few versions of the same large pool of packages. The selection is currently what I'm complaining about; we want stability *plus* the flexibility to deviate from this stability when we want to. Sticking to stable versions helps, until a new stable version is released. Then all the old stuff suddenly starts using the *new* stable version, and probably break. You must have far worse experiences than I have. My components usually work across many releases with only very rare need to intervene (Five twice broke one of my products; I think that almost was it). I think the problem will be reduced if you stick to stable releases. It won't go away: when a new major release of even one of the dependencies comes along, things might break. I want to avoid the whole situation of things breaking that used to work. Anyway, I think we're talking about two different topics: * better release management * my pet peeve: if I do release something today, it should work tomorrow, no matter what new releases happen of the dependencies. This should also work for frameworks where people build code on the framework I release. Regards, Martijn ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
Hey, On 9/27/07, Brian Sutherland [EMAIL PROTECTED] wrote: [snip] There is one I thought of, but it's a bit backwards. Essentially, Debian has a repository of mostly unmodified original egg tarballs. And, they've already done the hard work of maintaining sane dependencies. So, why not simply re-name the .orig.tar.gz in a Debian release repository to their original names and you have a working set corresponding to that release. If you add your selected personal working set, then you have a basis for working without bleeding. Does it contain the Zope 3.4 eggs? Will it have Zope 3.5 eggs when we need them? I think if we're going to manage our own gated community, it'll be something we need to do ourselves. I just see the need for 1 per release of piece of software (grok 0.10, grok 0.11, etc), and only a vague idea how frameworks would work (I want to use the Foo gated community which also uses the Grok gated community) so I must be missing something. Regards, Martijn ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
Martijn Faassen wrote at 2007-9-25 19:57 +0200: ... If you choose for flexibility first, people will need to think about versions all the time. I follow Tres argumentation: somehow the Linux distributors have this problem mostly solved: Standard distributions come with a set of known working components and quite weak dependancy declarations. When I install additional components, I will be told about potential conflicts (based on the weak dependancies) and asked what to do (install nevertheless, upgrade more things to get dependancies right or abort). Usually, I do not have to worry about versions -- only occationally (when I see conflict lists) or even more rarely, when something breaks even though there has not been a conflict. We currently made bad experiences with weak dependancies. I see several reasons for this: * not yet ready distributions (insufficiently tested, alpha quality) have been uploaded to PyPI We are now aware that we must not do things like this * installation tools have prefered newer versions over older ones, even when the newer versions were development/alpha while the older ones have been stable The tools meanwhile have changed and stick preferably to stable versions * The installation tools work incrementally with dependancies rather than based on a global dependancy graph. This is not yet changed. Maybe, our bad experience are drastically reduced when the above reasons are taken care of -- even with weak dependancies? -- Dieter ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
Dieter Maurer wrote: Martijn Faassen wrote at 2007-9-25 19:57 +0200: ... If you choose for flexibility first, people will need to think about versions all the time. I follow Tres argumentation: somehow the Linux distributors have this problem mostly solved: While I don't dispute we should look at package management systems, they don't have *our* problems exactly solved. I have 20 different buildouts installed, which may or may use the same pool of eggs, and may use different versions of the same package. I am the one who wants to have the final say in what versions of packages. I want to use. A linux distributor needs to have one working set of packages, instead. Standard distributions come with a set of known working components and quite weak dependancy declarations. When I install additional components, I will be told about potential conflicts (based on the weak dependancies) and asked what to do (install nevertheless, upgrade more things to get dependancies right or abort). Usually, I do not have to worry about versions -- only occationally (when I see conflict lists) or even more rarely, when something breaks even though there has not been a conflict. A linux distribution, unless you're on debian unstable, has quite a strict control over what packages enter a common pool. They do not release a newer version of a library unless they know the software that depends on it either works or can be upgraded to it. We have a situation where we have developers, not maintainers, uploading new versions of packages. There will be no integrated testing done for all software built on all packages in the cheeseshop. Again, I can see similarities, but I don't believe linux distributions have *exactly* our problems solved. Our buildouts are used as development environments, not only deployment environments. We currently made bad experiences with weak dependancies. I see several reasons for this: * not yet ready distributions (insufficiently tested, alpha quality) have been uploaded to PyPI We are now aware that we must not do things like this Better diligence here would help. It would help me most as a framework developer developing the framework - I can upgrade to newer versions of my dependencies without so much stuff breaking. * installation tools have prefered newer versions over older ones, even when the newer versions were development/alpha while the older ones have been stable The tools meanwhile have changed and stick preferably to stable versions Sticking to stable versions helps, until a new stable version is released. Then all the old stuff suddenly starts using the *new* stable version, and probably break. * The installation tools work incrementally with dependancies rather than based on a global dependancy graph. This is not yet changed. Maybe, our bad experience are drastically reduced when the above reasons are taken care of -- even with weak dependancies? I used to believe that, but after seeing Grok 0.10 break once or twice *every* week for the last month, I don't believe it anymore. We're talking about the same release, breaking over and over again as something goes wrong with some egg somewhere. I want these dependencies pinned down hard. That said, I believe there are ways to solve these problems without hardcoding them in install_requires. I hope we can have the benefits of weak dependencies while having the safetly of hardcoded ones. See here: http://faassen.n--tree.net/blog/view/weblog/2007/09/26/0 Regards, Martijn ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
On Sep 25, 2007, at 10:04 AM, Martijn Faassen wrote: Tres Seaver wrote: [snip] This is certainly an interesting approach. I'd be curious how you would garden this known working set. Martijn makes a pretty good case for maintaining such working sets close to the package in question (e.g. the grok egg, the Plone egg, etc.). I would argue that this problem is too big for developer convenience to drive it: we need concerted effort from the different communities of interest to manage the problem, in much the same way that Debian / Fedora etc. manage their various distribution releases. I want the ability to make releases of Grok so that when someone writes an application against it, it will continue to work, no matter what egg releases follow. That's a community of interest, I guess? I just want the technical ability to do so, and easily. If it's *not* convenient for developers to maintain it, it won't get maintained well by the various communities of interest. I think it's important to maintain these lists of dependencies close to what they talk about, preferably inside the packages themselves. As far as I know what is required is the ability for grok, in its setup.py, to include a list of suggested pinned dependencies (besides, and separate from, the normal dependencies). It should also be easy to configure buildout to inspect this list. What is also required is a way to easily create and maintain this list. Now I think you're talking about ways to maintain, report and test such lists below, but I want to solve my immediate problems first. I also need this solved preferably today. It's the primary hurt of Grok today. Everything else is peanuts. Of course, if we don't need flexibility to allow application developers to diverge from Grok's recommendation, this problem is solved today, except for the bit to actually generate the list of dependencies. We can simply hardcode them as dependencies in Grok's setup.py and tell everybody who wants to use newer versions of eggs for whatever reasons in their applications too bad, wait for the new release. I'm skeptical that you are going to get these changes in setuptools. I'm not crazy about them myself as they make writing setup files even more complicated. I'd much rather have a way for a comsumer to say: Use version V of project P even though it doesn't satisfy a dependency. Basically, a way to override a dependency. This is something that buildout could be taught to do, although *I* don't have time to do it today. OTOH, buildout already provides an alternate solution to this, which isn't good enough because you want something that will work w/o buildout. Oh well. JIm -- Jim Fulton Zope Corporation ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
On Sep 25, 2007, at 11:22 AM, Martijn Faassen wrote: Hey, On 9/25/07, Jim Fulton [EMAIL PROTECTED] wrote: [snip] I'm skeptical that you are going to get these changes in setuptools. I'm not crazy about them myself as they make writing setup files even more complicated. I'd much rather have a way for a comsumer to say: Use version V of project P even though it doesn't satisfy a dependency. Basically, a way to override a dependency. This is something that buildout could be taught to do, although *I* don't have time to do it today. That solves the problem too. Should we then encourage everyone to hardcode version numbers in their setup.py's dependencies list? If not, framework packages can maintain such lists. No, only in application (or very thick framework packages). This should certainly not be done for library packages. OTOH, buildout already provides an alternate solution to this, which isn't good enough because you want something that will work w/o buildout. Oh well. I am not sure what buildout mechanism you're referring to. A buildout configuration can specify a versions section that specified the versions to be used in the buildout. Other buildout configurations can extend the original configuration, specifying different versions as desired., A version list over HTTP? Configurations can be specified as URLs. Anything that urrlib2 understand will work. For example, you can include URLs in a buildout extends option. I also don't understand why you say I'm asking for it to work without buildout. You aren't? Cool. Well, other people have. When this issue first came up, Philipp worked out the buildout solution and others complained they wanted something that would work with workingenv, I'm not saying that this is an invalid desire. I'm just pointing out that there is a solution of you're willing to use buildout. The requirement is that not the application developers but the framework developers are easily able to maintain this list of dependencies. That is: * if someone puts the depencency to framework 1.7 in their dependencies in setup.py, it'll automatically get the pinned dependencies of this framework. Well, the buildout solution won't help that. * if someone tells us, we used framework 1.7, we *know* what dependencies this implied (unless they did a manual override). If reusing the framework includes reusing a buildout configuration that specifies the versions used by the framework, then buildout can help. So, if I write an application that uses framework 1.7, all I should have to do is set this in my dependencies list in setup.py. If I then decide to upgrade to 1.8, that's all I need to adjust. There is no long list of dependencies for me to maintain, as the framework developers do this for me. It'd be nice if I had the ability to override what the framework said, of course. setuptools doesn't let you do this. Maybe you want to lobby for a change on the distutils sig. Buildout could be enhanced to provide this feature, as I mentioned before. I know you referred us maintaining a buildout versions list on some HTTP site. This indeed solves most of the above. I don't think this ideal: * it doesn't work in a train. Of course, egg downloads don't either, but at least one tends to have a well-filled eggs directory already. * it requires the developers to upload a new list to some HTTP site each time they make a framework release. With eggs, it's nice that a few setup commands are all you need to do to make a release. So, I'd prefer to maintain this list in the package the list is talking about, and to only have a single release artifact (the package itself), instead of two (the package and the list). Of course, since it works now, we might want to do this for now anyway. I just don't think it's ideal. Agreed. Would it be possible for buildout to retrieve such a list from an egg if it's maintained as an extra, made-up setup.py variable? I just tried this, but it doesn't seem to be stored in egg-info. In theory, but not today. Your use case would be better served by adding a way to tell buildout to use a package version even if it violates some requirements, Finally, it would be nice to have the ability to maintain multiple lists of dependencies of significant sub-frameworks, and to have the ability to combine them in larger frameworks. That is, someone could be maintaining a Zope3-core package that depends on a whole bunch of Zope 3 packages, and Grok would then depend on this. I'd be nice if Grok wouldn't need to maintain its own list of pinned Zope3-core packages in this case but could rely on the release management and integrated testing procedure of the package it depends on. This is all would be nice right now, though. I think there are lots of potential complexity that all this flexibility could add. I'm not sure how nice that would be. In any case, you should probably raise this issue on
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
Martijn Faassen wrote at 2007-9-25 17:22 +0200: ... Should we then encourage everyone to hardcode version numbers in their setup.py's dependencies list? I think this goes against building applications from components -- as it drastically increases the probability of conflicts. Components should are week dependancy requirements to be maximally usable -- not fixed dependancies. I think, this holds for frameworks, too, as they are also components. -- Dieter ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Known working sets II [was: Eggification redux]
Hey, On 9/25/07, Dieter Maurer [EMAIL PROTECTED] wrote: Martijn Faassen wrote at 2007-9-25 17:22 +0200: ... Should we then encourage everyone to hardcode version numbers in their setup.py's dependencies list? I think this goes against building applications from components -- as it drastically increases the probability of conflicts. Components should are week dependancy requirements to be maximally usable -- not fixed dependancies. I think, this holds for frameworks, too, as they are also components. I understand this point of view, and I appreciate it. But on the other hand, if someone uses a framework, it should continue to work tomorrow. If someone releases a framework, it should remain installable tomorrow. If someone communicates to someone else about framework versions (important in open source software!), they shouldn't have to give a list of 50 version numbers, but just one. We therefore have two different requirements pulling on our system. On the one hand you don't want to lose flexibility, on the other hand if you want to have a community working and reusing chunks of software, you want to be able to rely on stability, and frequently you even want to count on bug for bug compatibility. If you choose to put the dependencies in lightly, you choose for flexibility. If then need to pin versions somewhere else. If you choose to pin strict dependencies right away, you choose for stability. You then need to have a way to override such dependency relationships somewhere else. The advantage of choosing for stability first is that people can install your software without the need to know details about versions. They only need to start thinking about it when they arrive at a reason to explicitly break them. If you choose for flexibility first, people will need to think about versions all the time. Regards, Martijn ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
