Re: [Zope3-dev] Re: RFC: Known working sets
Jim Fulton wrote: On Sep 5, 2007, at 10:48 AM, Chris Withers wrote: Jim Fulton wrote: I'm very much against making setuptools *more* complicated than it already is. Indeed, but surely managing known good sets of components comes under its remit of version management? Sure. It does this via requirements. Ok, forgive me for being dumb then, but why are we looking to add similar to zc.buildout? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Chris Withers wrote: Jim Fulton wrote: On Sep 5, 2007, at 10:48 AM, Chris Withers wrote: Jim Fulton wrote: I'm very much against making setuptools *more* complicated than it already is. Indeed, but surely managing known good sets of components comes under its remit of version management? Sure. It does this via requirements. Ok, forgive me for being dumb then, but why are we looking to add similar to zc.buildout? We're talking more about a general pattern in zc.buildout. The deficiencies of using setup.py for this alone are described well in the original proposal. Martin -- Acquisition is a jealous mistress ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
Martin Aspeli wrote: Jim Fulton wrote: I'm very much against making setuptools *more* complicated than it already is. Indeed, but surely managing known good sets of components comes under its remit of version management? Sure. It does this via requirements. Ok, forgive me for being dumb then, but why are we looking to add similar to zc.buildout? We're talking more about a general pattern in zc.buildout. The deficiencies of using setup.py for this alone are described well in the original proposal. Yup, and this was the reason for my original question to Jim: why do something in zc.buildout rather than fixing the problems with setuptools? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
On 9/6/07, Chris Withers [EMAIL PROTECTED] wrote: Yup, and this was the reason for my original question to Jim: why do something in zc.buildout rather than fixing the problems with setuptools? It's not at all clear to me that this suggests there's actually a problem with setuptools. The desire to express and work with known-good working sets does not suggest that setuptools is insufficient. We're interested in determining how we want to use setuptools for this, which may or may not turn up insufficiencies, but we've not identified any insufficiencies at this time. -Fred -- Fred L. Drake, Jr.fdrake at gmail.com Chaos is the score upon which reality is written. --Henry Miller ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
On Sep 6, 2007, at 4:02 AM, Chris Withers wrote: Martin Aspeli wrote: Jim Fulton wrote: I'm very much against making setuptools *more* complicated than it already is. Indeed, but surely managing known good sets of components comes under its remit of version management? Sure. It does this via requirements. Ok, forgive me for being dumb then, but why are we looking to add similar to zc.buildout? We're talking more about a general pattern in zc.buildout. The deficiencies of using setup.py for this alone are described well in the original proposal. Yup, and this was the reason for my original question to Jim: why do something in zc.buildout rather than fixing the problems with setuptools? Because neither the problem nor the fix are well understood, imo, and setuptools is already too complicated. Perhaps the same could be said about buildout, but no new buildout features are needed to experiment with the issue at this point. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: On Sep 6, 2007, at 4:02 AM, Chris Withers wrote: Martin Aspeli wrote: Jim Fulton wrote: I'm very much against making setuptools *more* complicated than it already is. Indeed, but surely managing known good sets of components comes under its remit of version management? Sure. It does this via requirements. Ok, forgive me for being dumb then, but why are we looking to add similar to zc.buildout? We're talking more about a general pattern in zc.buildout. The deficiencies of using setup.py for this alone are described well in the original proposal. Yup, and this was the reason for my original question to Jim: why do something in zc.buildout rather than fixing the problems with setuptools? Because neither the problem nor the fix are well understood, imo, and setuptools is already too complicated. Perhaps the same could be said about buildout, but no new buildout features are needed to experiment with the issue at this point. If we treat the subset as a catalog-side problem, then we don't need to change setuptools either: we can just use 'index-url' to point at the root of the subset. Perhaps PyPI can be extended to allow tag clouds, with sub-pages which select only packages / releases matching a given tag. E.g., I just created a small subset page to test this out: $ bin/easy_install --index-url=http://palladion.com/static/kgtest/ \ zope.testing Searching for zope.testing Reading http://palladion.com/static/kgtest/zope.testing/ Reading http://palladion.com/static/kgtest/zope.testing/3.4 Reading http://palladion.com/static/kgtest/zope.testing/3.0 Best match: zope.testing 3.4 Downloading \ http://pypi.python.org/packages/source/z/zope.testing/zope.testing-3.4.tar.gz#md5=5dbfed50da0169daff042da56f9bc439 Processing zope.testing-3.4.tar.gz Running zope.testing-3.4/setup.py -q bdist_egg --dist-dir \ /tmp/easy_install-kef7dg/zope.testing-3.4/egg-dist-tmp-EctfIO Adding zope.testing 3.4 to easy-install.pth file Installed \ /home/tseaver/tmp/foo/lib/python2.4/site-packages/zope.testing-3.4-py2.4.egg Processing dependencies for zope.testing Finished processing dependencies for zope.testing Another alternative would be to maintain the subset pages in SVN (perhaps with utilities for generating them from a manifest). Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG4BzH+gerLs4ltQ4RAuKGAJ9aB11PhcrB4uY/FQZTysyafr7z3QCfW78s AuLVzxJ8DZ+xh/KiS5dnbro= =Z4yF -END PGP SIGNATURE- ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
Jim Fulton wrote: I'm very much against making setuptools *more* complicated than it already is. Indeed, but surely managing known good sets of components comes under its remit of version management? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
On Sep 5, 2007, at 10:48 AM, Chris Withers wrote: Jim Fulton wrote: I'm very much against making setuptools *more* complicated than it already is. Indeed, but surely managing known good sets of components comes under its remit of version management? Sure. It does this via requirements. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Philipp von Weitershausen wrote: On 4 Sep 2007, at 01:21 , Tres Seaver wrote: Philipp von Weitershausen wrote: Wichert Akkerman wrote: The only problem is that distributing grok-0.11.cfg is a bit tedious. How about if buildout could get it from the web? How about making it available from an egg, through a hook in egg- info perhaps? This is a very good point. So good in fact that I thought of it myself :) I was already writing the email when your message came in. Martijn and I discussed the known working set problem over IRC this afternoon. He brought up a few good points which suggest that the version data could be associated with the egg: The approach that I described in my original posting (and which actually works *today*, as I found out!) has some disadvantages. For instance, the discoverability and release mechanism of the known working set configuration file differs a lot from our normal packages. Releasing a package is a well-known routine by now. How and where would we release the working set configuration file? SVN? Why not just release a meta-egg with hard-wired dependencies, plus the scripts required to set up and run the application? If the other components are as relaxed as possible about their dependencies, then it shouldn't matter that the meta-egg nails them down: it isn't going to be possilbe to re-use the meta-egg anyway (and I for one wouldn't want to -- I'd be creating a new environment to run it in). I think the use case of being able to override a particular version of a package is valid. Let's say I'm on Zope 3.5.0 (meta egg or whatever) and I'd like to get the newest ZODB because it has a really cool feature. Sure, the warranty is void now, but I'd still like to be able to do it *without* having to reproduce all the version dependencies that the meta egg specifies. Another problem are dependencies and how we'd like to depend on other working sets. Let's say we made a 'Zope' working set that contained the stable versions of the zope.* packages. It would make sense for grok to depend on this information. And packages using grok should depend on that. It'll be complicated to maintain such a chain in separate text files, especially across version updates. It only seems natural to use the egg dependency mechanism for this. Depending on another WS would basically Just Work (TM) if we used egg dependencies. It would, but setuptools doesn't allow us to specify overrides in case of version conflicts. Perhaps zc.buildout could be taught how. Then I suppose I could settle for using '==' in a meta egg's setup.py. When EGG-INFO is written, a custom writer will then take this information and generate 'known_working_versions.txt' or whatever in EGG-INFO. The only problem is that this custom writer needs to be installed when setup.py is called to create egg, in other words to generate the right kind of eggs you'd need to have something installed first. Not sure if this is better than maintaining .egg-info directories in SVN... I guess I don't get the usecase for maintaining known good dependencies outside of setup.py (for the meta egg). I don't mind maintaining them in setup.py, but the install_requires mechanism is insufficient. Sooner or later you're going to end up with a version conflict. This is definitely a case where looking at how the Linux distros packaging works is instructive: if you want to modify a package's dependencies (your overried, in this case), then you need to roll a new package. At that point, perhaps we need a tool which introspects a base egg's dependencies and creates a new set, overriding only what is different: in this case, the new egg would not depend on the old (in fact, it should *conflict with* or *replace* it). Linux distros take an approach that does not fit in the python world though: their meta-set is a release with its own package database. In other words every distribution/meta-set has its own PyPI instance database. Wichert. -- Wichert Akkerman [EMAIL PROTECTED] It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
Philipp von Weitershausen wrote: As far as I understand your use case, i twould already be covered by my original proposal: you always have the option to locally override what's specified in the working set. I think Dieter may have meant something like: [grok-0.11] grok = 0.11 ZODB = 3.8.0-3.8.4 zope.component = 3.4.0,3.5.1,3.5.2 I'm still a bit disappointed that we can't get this kind of support from distutils. cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
On 4 Sep 2007, at 10:15 , Chris Withers wrote: Philipp von Weitershausen wrote: As far as I understand your use case, i twould already be covered by my original proposal: you always have the option to locally override what's specified in the working set. I think Dieter may have meant something like: [grok-0.11] grok = 0.11 ZODB = 3.8.0-3.8.4 zope.component = 3.4.0,3.5.1,3.5.2 Ah. Interesting. I hadn't considered ranges before. It *does* make the whole algorithm more complicated, though. I'm still a bit disappointed that we can't get this kind of support from distutils. The question is whether that's a job for distutils / setuptools or not. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
On Sep 3, 2007, at 4:13 PM, Philipp von Weitershausen wrote:
Wichert Akkerman wrote:
The only problem is that distributing grok-0.11.cfg is a bit
tedious. How about if buildout could get it from the web?
How about making it available from an egg, through a hook in egg-info
perhaps?
This is a very good point. So good in fact that I thought of it
myself :) I was already writing the email when your message came in.
Martijn and I discussed the known working set problem over IRC this
afternoon. He brought up a few good points which suggest that the
version data could be associated with the egg:
The approach that I described in my original posting (and which
actually works *today*, as I found out!) has some disadvantages.
For instance, the discoverability and release mechanism of the
known working set configuration file differs a lot from our normal
packages. Releasing a package is a well-known routine by now. How
and where would we release the working set configuration file? SVN?
Another problem are dependencies and how we'd like to depend on
other working sets. Let's say we made a 'Zope' working set that
contained the stable versions of the zope.* packages. It would make
sense for grok to depend on this information. And packages using
grok should depend on that. It'll be complicated to maintain such a
chain in separate text files, especially across version updates. It
only seems natural to use the egg dependency mechanism for this.
So, a possible solution is to associate the known working version
info with an egg. More to the point, an egg could -- in addition to
simply listing the names of its dependencies -- also specify which
versions of those eggs it works best with. Wichert and I suggest
that this could be put in a file in the EGG-INFO directory.
The only problem is that we usually don't version control egg-info
directories. That means the information needs to be contained or at
least referenced in setup.py and then written upon setup.py
egg_info. Here's what setup.py *could* look like::
known_working_versions = {
'ZODB3': '3.8.0',
'zope.component': 3.4.0,
...
}
setup(
name='grok',
version='0.11',
...
install_requires=known_working_versions.keys(),
known_working_versions=known_working_versions
)
When EGG-INFO is written, a custom writer will then take this
information and generate 'known_working_versions.txt' or whatever
in EGG-INFO. The only problem is that this custom writer needs to
be installed when setup.py is called to create egg, in other words
to generate the right kind of eggs you'd need to have something
installed first. Not sure if this is better than maintaining .egg-
info directories in SVN...
How would the known_working_versions be used? You haven't
specified that. Presumably this would be something that is
overridable. How would this be overridden?
I'm very much against making setuptools *more* complicated than it
already is.
Perhaps buildout (and setuptools) should grow a mechanism for being
able to override/resolve version conflicts.
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED]Python
Powered!
CTO (540) 361-1714
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
Chris Withers wrote at 2007-9-4 09:15 +0100: Philipp von Weitershausen wrote: As far as I understand your use case, i twould already be covered by my original proposal: you always have the option to locally override what's specified in the working set. I think Dieter may have meant something like: [grok-0.11] grok = 0.11 ZODB = 3.8.0-3.8.4 zope.component = 3.4.0,3.5.1,3.5.2 I think this orthogonal description may not be strict enough -- as the testing effort grows exponentially. Usually, you have tested a few sets, say a single one grok 0.11, ZODB 2.8.9, zope.component = 2.4.0 Now, I must integrate grok 0.11 with a component that requires ZODB 2.9.0 and zope.component 2.5.2. I run the tests and can then say: grok 0.11 works also with ZODB 2.9.0 and zope.component 2.5.2. But I do not know (without explicit testing) that grok 0.11, ZODB 2.8.9, zope.component 2.5.2 is a working set as well. Thus, the knowledge based on the two tests cannot be expressed by grok = 0.11 ZODB = 2.8.9, 2.9.0 zope.component = 2.4.0, 2.5.2 -- Dieter ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dieter Maurer wrote: Wichert Akkerman wrote at 2007-9-4 09:12 +0200: ... Linux distros take an approach that does not fit in the python world though: their meta-set is a release with its own package database. In other words every distribution/meta-set has its own PyPI instance database. Not sure that I understand this correctly. If I do, then focus on a single distribution, e.g. Debian edge. This should be similar to the one PyPI database. Now, consider a .deb package, similar to one in the Debian distro but not identical. I assume that is about Tres szenario... Perhaps we need to think of the known good set as a PyPI subset, which is maintained in the same way that the various Debian distro lists are: they may end up pulling packages from the same pool, but they don't expose versions / packages which have not been opted in to the set. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG3Z7u+gerLs4ltQ4RAk6GAJ9A9RjMrUvylKGzJjezDuPa+u6SagCgjwXE u4GRIDX7tI5HnM8Q5utvRJU= =qi21 -END PGP SIGNATURE- ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
On 4 Sep 2007, at 20:07 , Tres Seaver wrote: Dieter Maurer wrote: Wichert Akkerman wrote at 2007-9-4 09:12 +0200: ... Linux distros take an approach that does not fit in the python world though: their meta-set is a release with its own package database. In other words every distribution/meta-set has its own PyPI instance database. Not sure that I understand this correctly. If I do, then focus on a single distribution, e.g. Debian edge. This should be similar to the one PyPI database. Now, consider a .deb package, similar to one in the Debian distro but not identical. I assume that is about Tres szenario... Perhaps we need to think of the known good set as a PyPI subset, which is maintained in the same way that the various Debian distro lists are: they may end up pulling packages from the same pool, but they don't expose versions / packages which have not been opted in to the set. That seems to suggest you would put the definition of the working set in the hands of the package index. In other words, give setuptools or zc.buildout a stripped down view of PyPI in which only the working set shows up. The question is how to maintain this. You'd really want to maintain this within an egg. So PyPI would have to expose the dependency information (I don't know if it does that over the XMLRPC API) so that you could build the working set information from that. The advantage of such an approach would be that it would work with existing tools. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: RFC: Known working sets
On 4 Sep 2007, at 16:13 , Jim Fulton wrote: How would the known_working_versions be used? You haven't specified that. You're right, I forgot that. In buildout.cfg, you'd then say: [buildout] versions = egg:grok==0.11 which would load the grok 0.11 egg before doing anything else, inspect its known working versions data and then do the rest of the work. Presumably this would be something that is overridable. How would this be overridden? Overridability would work like it does now, a la: [buildout] versions = egg:grok==0.11, version_overrides [version_overrides] ZODB3 = 3.9.0 I'm very much against making setuptools *more* complicated than it already is. Sounds reasonable. Perhaps buildout (and setuptools) should grow a mechanism for being able to override/resolve version conflicts. That might be enough already. Then we could use meta eggs to express working sets and the buildout versions=... mechanism to override those versions. It should also be possible to resolve a version conflict between, say, the meta egg (the working set) and a particular package you're using by letting zc.buildout (or setuptools) know which dependencies take precendence, e.g.:: [buildout] version_precedence = MyApp grok Zope which would mean that MyApp's dependencies take precendence over grok's dependencies which again take precendence over the Zope meta egg's dependencies. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Philipp von Weitershausen wrote: Deliverables * zc.buildout's extends mechanism would have to be enhanced to be able to load files from HTTP locations. Apparently this already works, I just tested it. I did not bother trying this out before because I didn't think buildout already supported it. Yay! -- http://worldcookery.com -- Professional Zope documentation and training ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Philipp von Weitershausen wrote: Philipp von Weitershausen wrote: Deliverables * zc.buildout's extends mechanism would have to be enhanced to be able to load files from HTTP locations. Apparently this already works, I just tested it. I did not bother trying this out before because I didn't think buildout already supported it. Yay! That's quite cool, but I still see a couple of downsides with this. - It only works through buildout. Ideally it would be supported at the setuptools level, imho. - I worry that the management of lots .cfg files could be cumbersome. For Plone, we could probably generate one from the dist_plone package, which otherwise lists known working sets for installers and plone.recipe.plone. - This doesn't really solve the dependency problem. I can't say, Give me Plone 3.0.1 or newer in a third party package. That could probably done by having a separate Plone meta-egg which uses = type dependency specifications, though. All in all, it looks like a workable solution, though. I'd like to try it out in practice, certainly. Martin -- Acquisition is a jealous mistress ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Wichert Akkerman wrote:
The only problem is that distributing grok-0.11.cfg is a bit tedious.
How about if buildout could get it from the web?
How about making it available from an egg, through a hook in egg-info
perhaps?
This is a very good point. So good in fact that I thought of it myself
:) I was already writing the email when your message came in.
Martijn and I discussed the known working set problem over IRC this
afternoon. He brought up a few good points which suggest that the
version data could be associated with the egg:
The approach that I described in my original posting (and which actually
works *today*, as I found out!) has some disadvantages. For instance,
the discoverability and release mechanism of the known working set
configuration file differs a lot from our normal packages. Releasing a
package is a well-known routine by now. How and where would we release
the working set configuration file? SVN?
Another problem are dependencies and how we'd like to depend on other
working sets. Let's say we made a 'Zope' working set that contained the
stable versions of the zope.* packages. It would make sense for grok to
depend on this information. And packages using grok should depend on
that. It'll be complicated to maintain such a chain in separate text
files, especially across version updates. It only seems natural to use
the egg dependency mechanism for this.
So, a possible solution is to associate the known working version info
with an egg. More to the point, an egg could -- in addition to simply
listing the names of its dependencies -- also specify which versions of
those eggs it works best with. Wichert and I suggest that this could be
put in a file in the EGG-INFO directory.
The only problem is that we usually don't version control egg-info
directories. That means the information needs to be contained or at
least referenced in setup.py and then written upon setup.py egg_info.
Here's what setup.py *could* look like::
known_working_versions = {
'ZODB3': '3.8.0',
'zope.component': 3.4.0,
...
}
setup(
name='grok',
version='0.11',
...
install_requires=known_working_versions.keys(),
known_working_versions=known_working_versions
)
When EGG-INFO is written, a custom writer will then take this
information and generate 'known_working_versions.txt' or whatever in
EGG-INFO. The only problem is that this custom writer needs to be
installed when setup.py is called to create egg, in other words to
generate the right kind of eggs you'd need to have something installed
first. Not sure if this is better than maintaining .egg-info directories
in SVN...
--
http://worldcookery.com -- Professional Zope documentation and training
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Martin Aspeli wrote: - It only works through buildout. Ideally it would be supported at the setuptools level, imho. I'm not really convinced that that's necessary. From a practical perspective, zc.buildout is the defacto deployment tool in the Zope community. Also, working sets have deployment written all over it. setuptools has little to no machinery to aid automated deployments in any way. That said, Wichert's proposed solution to put the information in EGG-INFO would make it open for use by other deployment tools, even setuptools. - I worry that the management of lots .cfg files could be cumbersome. For Plone, we could probably generate one from the dist_plone package, which otherwise lists known working sets for installers and plone.recipe.plone. Well, or the other way around: the installers could take the .cfg file of the working set and grab the right packages according to that. After all, a .cfg file can be read with ConfigParser quite easily. - This doesn't really solve the dependency problem. I can't say, Give me Plone 3.0.1 or newer in a third party package. That could probably done by having a separate Plone meta-egg which uses = type dependency specifications, though. Yes, this could be covered by a Plone egg (meta or not) and a Plone=3.0 modifier that's put in the 3rd party package's setup.py (I think the =, = operators can be valid in setup.py, just == isn't). -- http://worldcookery.com -- Professional Zope documentation and training ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Hi Philipp, - It only works through buildout. Ideally it would be supported at the setuptools level, imho. I'm not really convinced that that's necessary. From a practical perspective, zc.buildout is the defacto deployment tool in the Zope community. Alas, not so for all Plone people: Some folks prefer workingenv's in old-fashioned Zope (2) instances, some people use instancemanager, some people still symlink into lib/python. Also, working sets have deployment written all over it. Good point. setuptools has little to no machinery to aid automated deployments in any way. True. However, I think it's also legitimate to want to depend on a complete working set, which is more in setuptools land. But I see no problem in solving this at a buildout level first. That said, Wichert's proposed solution to put the information in EGG-INFO would make it open for use by other deployment tools, even setuptools. True. - I worry that the management of lots .cfg files could be cumbersome. For Plone, we could probably generate one from the dist_plone package, which otherwise lists known working sets for installers and plone.recipe.plone. Well, or the other way around: the installers could take the .cfg file of the working set and grab the right packages according to that. After all, a .cfg file can be read with ConfigParser quite easily. Yeah, true. It's just that everything wants to be the one true place. In the Plone land, having to cater to non-buildout deployments may make that harder, but like you said - .cfg files are pretty neutral. - This doesn't really solve the dependency problem. I can't say, Give me Plone 3.0.1 or newer in a third party package. That could probably done by having a separate Plone meta-egg which uses = type dependency specifications, though. Yes, this could be covered by a Plone egg (meta or not) and a Plone=3.0 modifier that's put in the 3rd party package's setup.py (I think the =, = operators can be valid in setup.py, just == isn't). Right. Anyway, like I said - I'd like to see a working version of this approach; Plone could quite usefully use it, imho. Martin -- Acquisition is a jealous mistress ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Martin Aspeli wrote: - It only works through buildout. Ideally it would be supported at the setuptools level, imho. I'm not really convinced that that's necessary. From a practical perspective, zc.buildout is the defacto deployment tool in the Zope community. Alas, not so for all Plone people: Some folks prefer workingenv's in old-fashioned Zope (2) instances, some people use instancemanager, some people still symlink into lib/python. Well, then let me rephrase: I'm not looking for a solution that fits everybody. I'm looking for *a* solution. And I wouldn't mind if it worked well with zc.buildout :). Plus, a ConfigParser-style configuration file can be read by pretty much anything so I'm sure workingenv, instancemanager and what not can easily be enhanced if someone cared enough. And as said before, if the stuff is placed in EGG-INFO, it'll also be extremely easy to discover. Also, working sets have deployment written all over it. Good point. setuptools has little to no machinery to aid automated deployments in any way. True. However, I think it's also legitimate to want to depend on a complete working set, which is more in setuptools land. But I see no problem in solving this at a buildout level first. setuptools only gives us the ability to depend on eggs. Working sets as ar as I define them are pinned down version definitions of dependencies. This doesn't exist in setuptools yet. The fact that we can use its dependency mechanism as an analog is pure convenience. - I worry that the management of lots .cfg files could be cumbersome. For Plone, we could probably generate one from the dist_plone package, which otherwise lists known working sets for installers and plone.recipe.plone. Well, or the other way around: the installers could take the .cfg file of the working set and grab the right packages according to that. After all, a .cfg file can be read with ConfigParser quite easily. Yeah, true. It's just that everything wants to be the one true place. In the Plone land, having to cater to non-buildout deployments may make that harder, but like you said - .cfg files are pretty neutral. Yup. Non-buildout deployments, such as installers, could still read the .cfg file from EGG-INFO. - This doesn't really solve the dependency problem. I can't say, Give me Plone 3.0.1 or newer in a third party package. That could probably done by having a separate Plone meta-egg which uses = type dependency specifications, though. Yes, this could be covered by a Plone egg (meta or not) and a Plone=3.0 modifier that's put in the 3rd party package's setup.py (I think the =, = operators can be valid in setup.py, just == isn't). Right. Anyway, like I said - I'd like to see a working version of this approach; Plone could quite usefully use it, imho. I trust by this approach you mean the EGG-INFO approach? Because the stuff I proposed originally already works... -- http://worldcookery.com -- Professional Zope documentation and training ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Dieter Maurer wrote: Therefore, it might be helpful, if the known working set would not need to be a singleton (consists of just one element). Assume the following use case: I use grok to build one of my applications. I need a another component as well which conflicts with grok. I override the grok configuration to get the version required for the other component. I run the grok testsuite and can then report to the grok community: grok is working with version V of component C as well. This kind of information would be helpful for integrators. It need not necessary be automatically supported by buildout. As far as I understand your use case, i twould already be covered by my original proposal: you always have the option to locally override what's specified in the working set. But I do agree that it should be possible to rely on other working sets, e.g. Grok application develoeprs relying on the Grok working set which in turn relies on a Zope working set. -- http://worldcookery.com -- Professional Zope documentation and training ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
Philipp von Weitershausen wrote: I trust by this approach you mean the EGG-INFO approach? Because the stuff I proposed originally already works... Actually, I meant your proposal: I'd like to see a concrete example of it, and a workable process/policy for managing releases in this way; and then I'd like to bake that into plone.recipe.plone. :) Plone's release management stuff needs a bit of a re-jig to avoid some duplication/management overhead anyway. Martin -- Acquisition is a jealous mistress ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Philipp von Weitershausen wrote:
Wichert Akkerman wrote:
The only problem is that distributing grok-0.11.cfg is a bit tedious.
How about if buildout could get it from the web?
How about making it available from an egg, through a hook in egg-info
perhaps?
This is a very good point. So good in fact that I thought of it myself
:) I was already writing the email when your message came in.
Martijn and I discussed the known working set problem over IRC this
afternoon. He brought up a few good points which suggest that the
version data could be associated with the egg:
The approach that I described in my original posting (and which actually
works *today*, as I found out!) has some disadvantages. For instance,
the discoverability and release mechanism of the known working set
configuration file differs a lot from our normal packages. Releasing a
package is a well-known routine by now. How and where would we release
the working set configuration file? SVN?
Why not just release a meta-egg with hard-wired dependencies, plus the
scripts required to set up and run the application? If the other
components are as relaxed as possible about their dependencies, then it
shouldn't matter that the meta-egg nails them down: it isn't going to
be possilbe to re-use the meta-egg anyway (and I for one wouldn't want
to -- I'd be creating a new environment to run it in).
Another problem are dependencies and how we'd like to depend on other
working sets. Let's say we made a 'Zope' working set that contained the
stable versions of the zope.* packages. It would make sense for grok to
depend on this information. And packages using grok should depend on
that. It'll be complicated to maintain such a chain in separate text
files, especially across version updates. It only seems natural to use
the egg dependency mechanism for this.
Depending on another WS would basically Just Work (TM) if we used egg
dependencies.
So, a possible solution is to associate the known working version info
with an egg. More to the point, an egg could -- in addition to simply
listing the names of its dependencies -- also specify which versions of
those eggs it works best with. Wichert and I suggest that this could be
put in a file in the EGG-INFO directory.
The only problem is that we usually don't version control egg-info
directories. That means the information needs to be contained or at
least referenced in setup.py and then written upon setup.py egg_info.
Here's what setup.py *could* look like::
known_working_versions = {
'ZODB3': '3.8.0',
'zope.component': 3.4.0,
...
}
setup(
name='grok',
version='0.11',
...
install_requires=known_working_versions.keys(),
known_working_versions=known_working_versions
)
When EGG-INFO is written, a custom writer will then take this
information and generate 'known_working_versions.txt' or whatever in
EGG-INFO. The only problem is that this custom writer needs to be
installed when setup.py is called to create egg, in other words to
generate the right kind of eggs you'd need to have something installed
first. Not sure if this is better than maintaining .egg-info directories
in SVN...
I guess I don't get the usecase for maintaining known good
dependencies outside of setup.py (for the meta egg).
Tres.
- --
===
Tres Seaver +1 540-429-0999 [EMAIL PROTECTED]
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFG3Jb6+gerLs4ltQ4RAsnCAJ4gVDaVp3Q23zERjzexEVjDgYjf6gCfecSv
sf4f3BZxwVvqxgJ4FASX+3w=
=O0Ea
-END PGP SIGNATURE-
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: RFC: Known working sets
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Philipp von Weitershausen wrote: On 4 Sep 2007, at 01:21 , Tres Seaver wrote: Philipp von Weitershausen wrote: Wichert Akkerman wrote: The only problem is that distributing grok-0.11.cfg is a bit tedious. How about if buildout could get it from the web? How about making it available from an egg, through a hook in egg- info perhaps? This is a very good point. So good in fact that I thought of it myself :) I was already writing the email when your message came in. Martijn and I discussed the known working set problem over IRC this afternoon. He brought up a few good points which suggest that the version data could be associated with the egg: The approach that I described in my original posting (and which actually works *today*, as I found out!) has some disadvantages. For instance, the discoverability and release mechanism of the known working set configuration file differs a lot from our normal packages. Releasing a package is a well-known routine by now. How and where would we release the working set configuration file? SVN? Why not just release a meta-egg with hard-wired dependencies, plus the scripts required to set up and run the application? If the other components are as relaxed as possible about their dependencies, then it shouldn't matter that the meta-egg nails them down: it isn't going to be possilbe to re-use the meta-egg anyway (and I for one wouldn't want to -- I'd be creating a new environment to run it in). I think the use case of being able to override a particular version of a package is valid. Let's say I'm on Zope 3.5.0 (meta egg or whatever) and I'd like to get the newest ZODB because it has a really cool feature. Sure, the warranty is void now, but I'd still like to be able to do it *without* having to reproduce all the version dependencies that the meta egg specifies. Another problem are dependencies and how we'd like to depend on other working sets. Let's say we made a 'Zope' working set that contained the stable versions of the zope.* packages. It would make sense for grok to depend on this information. And packages using grok should depend on that. It'll be complicated to maintain such a chain in separate text files, especially across version updates. It only seems natural to use the egg dependency mechanism for this. Depending on another WS would basically Just Work (TM) if we used egg dependencies. It would, but setuptools doesn't allow us to specify overrides in case of version conflicts. Perhaps zc.buildout could be taught how. Then I suppose I could settle for using '==' in a meta egg's setup.py. When EGG-INFO is written, a custom writer will then take this information and generate 'known_working_versions.txt' or whatever in EGG-INFO. The only problem is that this custom writer needs to be installed when setup.py is called to create egg, in other words to generate the right kind of eggs you'd need to have something installed first. Not sure if this is better than maintaining .egg-info directories in SVN... I guess I don't get the usecase for maintaining known good dependencies outside of setup.py (for the meta egg). I don't mind maintaining them in setup.py, but the install_requires mechanism is insufficient. Sooner or later you're going to end up with a version conflict. This is definitely a case where looking at how the Linux distros packaging works is instructive: if you want to modify a package's dependencies (your overried, in this case), then you need to roll a new package. At that point, perhaps we need a tool which introspects a base egg's dependencies and creates a new set, overriding only what is different: in this case, the new egg would not depend on the old (in fact, it should *conflict with* or *replace* it). Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG3Ju4+gerLs4ltQ4RAmy0AJ4h8P9yQeRKqG7Qm3iuGN7e93yQrwCgkdOn PPe8f3YxJES0gbYG60/NqhI= =SIBv -END PGP SIGNATURE- ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
