[Zope-PAS] Re: Eggification redux
Charlie Clark wrote: Am 25.09.2007 um 02:05 schrieb Tres Seaver: I'd like to break the remaining CMF packages out (moving from '/CMF' to 'Products.CMFCore', 'Products.CMFDefault', etc.) and push the 2.1.0 eggs out, as well as equivalent changes for PluggableAuthService and PluginRegistry. Any objections, or other thoughts? While I am very sceptical about the move to eggs (I know it's inevitable) and I hope we can avoid some of the problems that seem to be affecting Grok as a result. Grok's problems are primarily related to the lack of a working set definition for Zope 3. We badly need it, not only for Zope 3 proper but also for projects which consume Zope 3 eggs (probably Zope 2 in the future, but naturally also grok). +1 to the eggification -- http://worldcookery.com -- Professional Zope documentation and training ___ Zope-PAS mailing list Zope-PAS@zope.org http://mail.zope.org/mailman/listinfo/zope-pas
Re: [Zope-PAS] Re: Eggification redux
On Sep 25, 2007, at 3:40 AM, Philipp von Weitershausen wrote: Charlie Clark wrote: Am 25.09.2007 um 02:05 schrieb Tres Seaver: I'd like to break the remaining CMF packages out (moving from '/ CMF' to 'Products.CMFCore', 'Products.CMFDefault', etc.) and push the 2.1.0 eggs out, as well as equivalent changes for PluggableAuthService and PluginRegistry. Any objections, or other thoughts? While I am very sceptical about the move to eggs (I know it's inevitable) and I hope we can avoid some of the problems that seem to be affecting Grok as a result. Grok's problems are primarily related to the lack of a working set definition for Zope 3. I don't know what you mean by this. Jim -- Jim Fulton Zope Corporation ___ Zope-PAS mailing list Zope-PAS@zope.org http://mail.zope.org/mailman/listinfo/zope-pas
[Zope-PAS] Re: Eggification redux
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: On Sep 25, 2007, at 3:40 AM, Philipp von Weitershausen wrote: Charlie Clark wrote: Am 25.09.2007 um 02:05 schrieb Tres Seaver: I'd like to break the remaining CMF packages out (moving from '/ CMF' to 'Products.CMFCore', 'Products.CMFDefault', etc.) and push the 2.1.0 eggs out, as well as equivalent changes for PluggableAuthService and PluginRegistry. Any objections, or other thoughts? While I am very sceptical about the move to eggs (I know it's inevitable) and I hope we can avoid some of the problems that seem to be affecting Grok as a result. Grok's problems are primarily related to the lack of a working set definition for Zope 3. I don't know what you mean by this. This is the known good problem. I'm pretty convinced that adding some kind of PyPI subset, where gardeners for a given package set keep the list of packages / versions known to work well together, is the only way out of this issue. E.g., a URL like: http://pypi.python.org/pypi/release/zope3.4 would be usable as an 'index-url' for setuptools: when used this way, setuptools would only find / install eggs from the gardened set, rather than whatever anyone happened to have uploaded that day. If PyPI can't be tweaked to provide such a feature, we may need to come up with some kind of mirroring scheme which does allow it. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG+PWv+gerLs4ltQ4RApd3AKCNANKTREE+1V7MJEexRgwNZl0DSwCfTTdL W5IG3LF2jjLbYnVTl1/6+YA= =GlfL -END PGP SIGNATURE- ___ Zope-PAS mailing list Zope-PAS@zope.org http://mail.zope.org/mailman/listinfo/zope-pas
Re: [Zope-CMF] Re: [Zope-PAS] Re: Eggification redux
Previously Philipp von Weitershausen wrote: On 25 Sep 2007, at 13:20 , Jim Fulton wrote: On Sep 25, 2007, at 3:40 AM, Philipp von Weitershausen wrote: Charlie Clark wrote: Am 25.09.2007 um 02:05 schrieb Tres Seaver: I'd like to break the remaining CMF packages out (moving from '/ CMF' to 'Products.CMFCore', 'Products.CMFDefault', etc.) and push the 2.1.0 eggs out, as well as equivalent changes for PluggableAuthService and PluginRegistry. Any objections, or other thoughts? While I am very sceptical about the move to eggs (I know it's inevitable) and I hope we can avoid some of the problems that seem to be affecting Grok as a result. Grok's problems are primarily related to the lack of a working set definition for Zope 3. I don't know what you mean by this. There are several problems. * We've had to nail some of the versions because buildout was being a bit over-zealous when getting eggs on Windows. It would take the newest egg, despite the fact that other eggs had binary releases. I guess that's not its fault. But it's a working set problem. * When package A depends on Y and package B also depends on Y, but with some version restrictions, buidout will first try to get the newest version of Y when installing A. But then when installing B, it is likely that it has to get a different version of Y. The result is a version conflict. This could also easily be fixed with a working set that dictates which versions would be used from the beginning. This is a real problem and something that has to be fixed. It will bite us with the separate repositories Tres mentioned as well. You need to be able to analyze the dependency/conflict graph before doing anything or you will risk breaking a working system. WIchert. -- Wichert Akkerman [EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-PAS mailing list Zope-PAS@zope.org http://mail.zope.org/mailman/listinfo/zope-pas
Re: [Zope-PAS] Re: Eggification redux
On Sep 25, 2007, at 7:55 AM, Philipp von Weitershausen wrote: On 25 Sep 2007, at 13:20 , Jim Fulton wrote: On Sep 25, 2007, at 3:40 AM, Philipp von Weitershausen wrote: Charlie Clark wrote: Am 25.09.2007 um 02:05 schrieb Tres Seaver: I'd like to break the remaining CMF packages out (moving from '/ CMF' to 'Products.CMFCore', 'Products.CMFDefault', etc.) and push the 2.1.0 eggs out, as well as equivalent changes for PluggableAuthService and PluginRegistry. Any objections, or other thoughts? While I am very sceptical about the move to eggs (I know it's inevitable) and I hope we can avoid some of the problems that seem to be affecting Grok as a result. Grok's problems are primarily related to the lack of a working set definition for Zope 3. I don't know what you mean by this. There are several problems. * We've had to nail some of the versions because buildout was being a bit over-zealous when getting eggs on Windows. It would take the newest egg, despite the fact that other eggs had binary releases. I guess that's not its fault. But it's a working set problem. It's probably a setuptools problem. It would probably make sense to give buildout a switch to prefer binary releases where they are available. (Perhaps the new prefer-final option would help here.) * When package A depends on Y and package B also depends on Y, but with some version restrictions, buidout will first try to get the newest version of Y when installing A. But then when installing B, it is likely that it has to get a different version of Y. The result is a version conflict. This could also easily be fixed with a working set that dictates which versions would be used from the beginning. IN the long run, this would be better served by a better algorithm for constructing setuptools working sets. BTW, since working set has a rather important meaning for setuptools, I'd rather we find a different term for what you are talking about. * Even with newest=false enabled in grokproject's buildout.cfg template, the versions that people will end up when trying out grok are non-deterministic. This has led to people installing newer versions of something, sometimes even faulty releases, even though Grok hadn't been tested with these newer versions yet and older versions that were known to work were the better choice. Again, a working set problem. Right, but I don't understand how having one of these things for Zope 3 would help. After all, a new package that is tested and added to the 3.4 release might still not work well with Grok. Jim -- Jim Fulton Zope Corporation ___ Zope-PAS mailing list Zope-PAS@zope.org http://mail.zope.org/mailman/listinfo/zope-pas
Re: [Zope-PAS] Re: Eggification redux
On Sep 25, 2007, at 8:13 AM, Philipp von Weitershausen wrote: On 25 Sep 2007, at 14:06 , Jim Fulton wrote: There are several problems. * We've had to nail some of the versions because buildout was being a bit over-zealous when getting eggs on Windows. It would take the newest egg, despite the fact that other eggs had binary releases. I guess that's not its fault. But it's a working set problem. It's probably a setuptools problem. It would probably make sense to give buildout a switch to prefer binary releases where they are available. (Perhaps the new prefer-final option would help here.) Not when it has to chose between ZODB 3.8.0b2 and 3.9.0a1dev- r12345, of which are neither a final release. We really need to get to final releases. Right. We really need to shorten ZODB's release cycle, or stop making new releases. :) * When package A depends on Y and package B also depends on Y, but with some version restrictions, buidout will first try to get the newest version of Y when installing A. But then when installing B, it is likely that it has to get a different version of Y. The result is a version conflict. This could also easily be fixed with a working set that dictates which versions would be used from the beginning. IN the long run, this would be better served by a better algorithm for constructing setuptools working sets. ... which would require having access to the dependency data before installation. No, not really, at least not in buildout's case. It's really not that big a deal to download a distribution that you ultimately don't use. I agree it might be better if the index made dependency data available. ... BTW, since working set has a rather important meaning for setuptools, I'd rather we find a different term for what you are talking about. Oh, you're right. Tres has used the term known good set in the past, I believe. * Even with newest=false enabled in grokproject's buildout.cfg template, the versions that people will end up when trying out grok are non-deterministic. This has led to people installing newer versions of something, sometimes even faulty releases, even though Grok hadn't been tested with these newer versions yet and older versions that were known to work were the better choice. Again, a working set problem. Right, but I don't understand how having one of these things for Zope 3 would help. After all, a new package that is tested and added to the 3.4 release might still not work well with Grok. I see known good sets like the old tarball release, in the sense that we can say Grok 0.x works with Zope 3.y.z (for specific x, y, z). I have a feeling that this won't be enough, but I'll defer. :) Likewise, I'd like to be able to say that Grok 0.x is known to work well with a particular known good set of Zope. Modifying the known good set (by adding a newer version of a package, or by adding a new package altogether) would constitute a new version of the known good set which we'd have to try Grok out with before we give it our blessing. If we apply our release semantics (major/minor/bugfix versions) to known working sets (I think we should), then we might also loosen the nails a bit and say that as long as you stay within the bugfix versions of one known good set, you're fine. We've done this for releases in the past, I don't see why we can't keep doing this. This works fine as long as nothing changes. I think you'd be better off keeping a grok kgs. But, again, I'll defer. :) Jim -- Jim Fulton Zope Corporation ___ Zope-PAS mailing list Zope-PAS@zope.org http://mail.zope.org/mailman/listinfo/zope-pas
Re: [Zope-PAS] Re: Eggification redux
On Sep 25, 2007, at 9:53 AM, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: * When package A depends on Y and package B also depends on Y, but with some version restrictions, buidout will first try to get the newest version of Y when installing A. But then when installing B, it is likely that it has to get a different version of Y. The result is a version conflict. This could also easily be fixed with a working set that dictates which versions would be used from the beginning. IN the long run, this would be better served by a better algorithm for constructing setuptools working sets. ... which would require having access to the dependency data before installation. No, not really, at least not in buildout's case. It's really not that big a deal to download a distribution that you ultimately don't use. If we can keep setuptools from prematurely installing downloaded eggs, sure; I'm not sure what you are saying. I maintain that downloading eggs you don't use is not that big a deal. Avoiding this is just an optimization. at the moment, it is trivial to get setuptools into a false conflict situaltion, because it processes dependencies incrementally, rather than solving the transitive closure of the graph before attempting to install anything. Sure, but this doesn't have anything to do with whether you download or not. It may be a consequence of the fact that setuptools tries to hard to figure out what to use without downloading enough. There is a similar problem with extras. I agree it might be better if the index made dependency data available. Not exposing the dependency information in the index seems like a missing-feature-is-really-a-bug to me. Likely. A problem is that there are 2 sigs when there really ought to be one. The distutils sig isn't demanding enough of the catalog- sig for changes to the index and the catalog-sig is somewhat grudglingly responsive. Some of this has to do with the fact that people who want catalog features really ought to help with implementation, but people who want the features don't have the time to learn the catalog app enough to enhance it. It would be nice if one of us was willing to become a catalog contributor so we could get some things done we need. Not it! :) Jim -- Jim Fulton Zope Corporation ___ Zope-PAS mailing list Zope-PAS@zope.org http://mail.zope.org/mailman/listinfo/zope-pas
