[Zope-dev] Re: Egg install bot results
Chris McDonough wrote: - Institute a policy that all distributions that are released to the cheeseshop should be installable via easy_install. IMO, if they are not installable this way, they should not be released to the cheeseshop, given the larger Python community's expectations. +1 - Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this. Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing. - Ditch the idea of releasing separate distributions for each package in zope.app. +1 The individual eggs typically can't be used outside of a zope appserver installation (and if they can, they probably shouldn't be in zope.app, they should be in zope or they should be their own top-level package), and the namespaciness of zope.app is suspect when it's unlikely that anyone who is not a Zope committer will release a distribution which makes use of that namespace package. Their current overgranularity makes distributing them as separate eggs and releasing them to the cheeseshop a form of cheeseshop pollution, especially given that so few of them can actually currently be installed using easy_install or setup.py install. If cheeseshop is going to continue to be used as the index, I'd suggest creating a zope.app top-level svn module with a single setup.py in containing all the packages that are meant to go into zope.app. Version the resulting zope.app distribution as necessary instead of versioning many more granular zope.app.* distros. It's OK if some people don't use some of the functionality in the resulting egg, just toss everything in. We can (and should) do that... almost :). For ages, the zope.app package has been treated as a namespace package because it always contained stuff that was releasable and stuff that wasn't (e.g. zope.app.homefolder or zope.app.css were experiments that probably shouldn't have been committed there in the first place, but alas, it happened). I don't think we should start releasing stuff that isn't in a releasable state. There is precedent here in the Paste distribution. It has many submodules and does many things, but it comes in the form of a single egg. Except that PasteScript and PasteDeploy are separate, which is just weird, especially considering the awkward naming of the eggs... Yes, you lose the ability to make a bugfix in one subpackage and release it, but IIRC the intent is to trim zope.app down anyway, pushing libraryish things out to top-level or zope.* packages. That *was* the plan at least. We moved several packages in the Zope 3.2 - 3.3 upgrade, but a lot of people complained, so we pretty much stopped doing that. Except when recently the Python parts of zope.app.securitypolicy, zope.app.session were moved to zope.securitypolicy, zope.session in an attempt to be able to reuse the Python stuff without getting the browser views. - Who's in charge? Whomever you might be, to what extent do you agree/disagree with the above suggestions? If you agree with any, how can I help fix things? - I'm unsure how anybody is able to install Zope 3 right now using eggs, unless there's some fundamental difference in the way easy_install resolves dependencies vs. buildout. I have not looked at how Stephan's KGS works yet, though, so I'm sure I'm just missing some magic. - We should consider fixing setuptools install to detect conflicts before attempting to install anything. +1 The current regime of find conflicts halfway through an install is IMO untenable in the long term for using eggs as a distribution mechanism. This may mean a very invasive change to both the package index and the client software, though. Yup. The problem is that people run custom indexes locally so setuptools would have to be able to deal with different kinds of index implementations. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Egg install bot results
On Wednesday 14 November 2007, Philipp von Weitershausen wrote: - Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this. Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing. They don't anymore. I removed the find-links in all the packages I touched while creating Zope 3.4.0b2, because distribution contains so many non-ready packages and would have overridden several of the KGS package versions. Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Egg install bot results
On Wednesday 14 November 2007, Philipp von Weitershausen wrote: Yes, you lose the ability to make a bugfix in one subpackage and release it, but IIRC the intent is to trim zope.app down anyway, pushing libraryish things out to top-level or zope.* packages. That *was* the plan at least. We moved several packages in the Zope 3.2 - 3.3 upgrade, but a lot of people complained, so we pretty much stopped doing that. Except when recently the Python parts of zope.app.securitypolicy, zope.app.session were moved to zope.securitypolicy, zope.session in an attempt to be able to reuse the Python stuff without getting the browser views. There are about 3-4 packages in zope.app with which we have to do the same, so that I can actually create a setup that does not have the ZMI, zope.formlib and zope.app.form. Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Egg install bot results
On Nov 14, 2007, at 6:05 AM, Stephan Richter wrote: On Wednesday 14 November 2007, Philipp von Weitershausen wrote: - Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this. Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing. They don't anymore. I removed the find-links in all the packages I touched while creating Zope 3.4.0b2, because distribution contains so many non-ready packages and would have overridden several of the KGS package versions. So what makes this stuff work via buildout is all version-pinning now? Assuming you agree that packages released to the cheeseshop should be installable via easy_install, if distribution dependencies are version-pinned in a distribution's buildout.cfg, the same pin should be done in the setup.py install_requires= for that distribution. Otherwise running easy_install will never have the same result as running buildout when it comes to fetching dependencies for a particular distribution. In general, buildout.cfg and setup.py should share all dependency information, or the version-pin in buildout.cfg should disappear. - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Egg install bot results
On Nov 14, 2007, at 8:38 AM, Chris McDonough wrote: On Nov 14, 2007, at 6:05 AM, Stephan Richter wrote: On Wednesday 14 November 2007, Philipp von Weitershausen wrote: - Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this. Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing. They don't anymore. I removed the find-links in all the packages I touched while creating Zope 3.4.0b2, because distribution contains so many non-ready packages and would have overridden several of the KGS package versions. So what makes this stuff work via buildout is all version-pinning now? They aren't using the cheeseshop. They are using the KGS instead. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Egg install bot results
On Nov 14, 2007, at 4:46 AM, Philipp von Weitershausen wrote: ... The individual eggs typically can't be used outside of a zope appserver installation (and if they can, they probably shouldn't be in zope.app, they should be in zope or they should be their own top-level package), and the namespaciness of zope.app is suspect when it's unlikely that anyone who is not a Zope committer will release a distribution which makes use of that namespace package. Their current overgranularity makes distributing them as separate eggs and releasing them to the cheeseshop a form of cheeseshop pollution, especially given that so few of them can actually currently be installed using easy_install or setup.py install. If cheeseshop is going to continue to be used as the index, I'd suggest creating a zope.app top-level svn module with a single setup.py in containing all the packages that are meant to go into zope.app. Version the resulting zope.app distribution as necessary instead of versioning many more granular zope.app.* distros. It's OK if some people don't use some of the functionality in the resulting egg, just toss everything in. We can (and should) do that... almost :). For ages, the zope.app package has been treated as a namespace package because it always contained stuff that was releasable and stuff that wasn't (e.g. zope.app.homefolder or zope.app.css were experiments that probably shouldn't have been committed there in the first place, but alas, it happened). If we put a lot of these packages back together, we should still treat zope.app as a namespace package. Let's not go back on that. One caution is that distutils and setuptools provide no help with multiple eggs that contain the same package. This means that someone can install the combined and separate packages at the same time (like they can now install ZODB3 and transaction) leading to subtle bugs. Probably the biggest mistake we made in this process was moving to fast. Whatever we do to fix the current situation, let's be very careful not to make it worse. I don't think we should start releasing stuff that isn't in a releasable state. You mean we never should have started. :) ... Yes, you lose the ability to make a bugfix in one subpackage and release it, but IIRC the intent is to trim zope.app down anyway, pushing libraryish things out to top-level or zope.* packages. That *was* the plan at least. We moved several packages in the Zope 3.2 - 3.3 upgrade, but a lot of people complained, so we pretty much stopped doing that. Except when recently the Python parts of zope.app.securitypolicy, zope.app.session were moved to zope.securitypolicy, zope.session in an attempt to be able to reuse the Python stuff without getting the browser views. IOW, we have to balance cleanliness with disruption. For a while, those of us working on Zope 3 revelled in refactor mercilessly. This was fine before Zope 3 was widely adopted. We have since leaned the value of mercy. :) - Who's in charge? Whomever you might be, to what extent do you agree/disagree with the above suggestions? If you agree with any, how can I help fix things? - I'm unsure how anybody is able to install Zope 3 right now using eggs, unless there's some fundamental difference in the way easy_install resolves dependencies vs. buildout. I have not looked at how Stephan's KGS works yet, though, so I'm sure I'm just missing some magic. - We should consider fixing setuptools install to detect conflicts before attempting to install anything. +1 I strongly suspect that this will be impractical. The current regime of find conflicts halfway through an install is IMO untenable in the long term for using eggs as a distribution mechanism. This may mean a very invasive change to both the package index and the client software, though. Yup. The problem is that people run custom indexes locally so setuptools would have to be able to deal with different kinds of index implementations. Huh? There is a defined index API. The custom indexes conform to this afaik. Things could certainly be made faster if the index provided access to more meta data, still resolving conflicts will still be potentially computationally very intensive. I still believe that a setuptools prefer-final switch would go a long way towards addressing this problem. I'll also not that, AFAIK, the current problems weren't due to true conflicts, but due to packages in the cheeseshop requiring package releases that weren't in the cheeseshop. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists -
Re: [Zope-dev] Re: Egg install bot results
On Nov 14, 2007, at 9:57 AM, Jim Fulton wrote: On Nov 14, 2007, at 8:38 AM, Chris McDonough wrote: On Nov 14, 2007, at 6:05 AM, Stephan Richter wrote: On Wednesday 14 November 2007, Philipp von Weitershausen wrote: - Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this. Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing. They don't anymore. I removed the find-links in all the packages I touched while creating Zope 3.4.0b2, because distribution contains so many non-ready packages and would have overridden several of the KGS package versions. So what makes this stuff work via buildout is all version-pinning now? They aren't using the cheeseshop. They are using the KGS instead. Yup. If I run: bin/easy_install --index-url=http://download.zope.org/zope3.4 zope.tales It works, good news. I sent a mail before (which didn't appear here, I'm not sure why not, at least I didn't get it back) saying that I was not comfortable making a new release of zope.app.publisher and/or zope.traversing to fix the recurring dependency problem exposed by the bot on the cheeseshop, nor do I have access to do it. Either this needs to get done or the bogus zope.app.publisher 3.5.0a2 release needs to get taken down in order to at least temporarily fix installation from the cheeseshop for arbitrary users. Is someone with familiarity of these packages and access to change those projects on the cheeseshop willing to do that? In the meantime, whose feet would I be shooting if I changed setup.py dependency info so that, e.g. zope.tales (and the like) didn't pull in as many dependencies? Installing zope.tales (as an example, I'm sure its as a result of a transitive dependency lower in the dep chain) right now gets us ~ 75 other distribution, including very unrelated ones: ./docutils-0.4-py2.4.egg ./pytz-2007g-py2.4.egg ./RestrictedPython-3.4.2-py2.4.egg ./ZConfig-2.5-py2.4.egg ./zdaemon-2.0.1-py2.4.egg ./ZODB3-3.8.0b4-py2.4-macosx-10.3-i386.egg ./zodbcode-3.4.0-py2.4.egg ./zope.annotation-3.4.0-py2.4.egg ./zope.app.applicationcontrol-3.4.1-py2.4.egg ./zope.app.appsetup-3.4.1-py2.4.egg ./zope.app.authentication-3.4.1-py2.4.egg ./zope.app.basicskin-3.4.0-py2.4.egg ./zope.app.broken-3.4.0-py2.4.egg ./zope.app.component-3.4.1-py2.4.egg ./zope.app.container-3.5.3-py2.4-macosx-10.3-i386.egg ./zope.app.content-3.4.0-py2.4.egg ./zope.app.debug-3.4.0-py2.4.egg ./zope.app.dependable-3.4.0-py2.4.egg ./zope.app.error-3.5.1-py2.4.egg ./zope.app.exception-3.4.1-py2.4.egg ./zope.app.folder-3.4.0-py2.4.egg ./zope.app.form-3.4.1-py2.4.egg ./zope.app.generations-3.4.1-py2.4.egg ./zope.app.http-3.4.1-py2.4.egg ./zope.app.i18n-3.4.4-py2.4.egg ./zope.app.interface-3.4.0-py2.4.egg ./zope.app.locales-3.4.0-py2.4.egg ./zope.app.pagetemplate-3.4.0-py2.4.egg ./zope.app.principalannotation-3.4.0-py2.4.egg ./zope.app.publication-3.4.2-py2.4.egg ./zope.app.publisher-3.4.1-py2.4.egg ./zope.app.renderer-3.4.0-py2.4.egg ./zope.app.rotterdam-3.4.1-py2.4.egg ./zope.app.schema-3.4.0-py2.4.egg ./zope.app.security-3.4.0-py2.4.egg ./zope.app.session-3.5.1-py2.4.egg ./zope.app.testing-3.4.1-py2.4.egg ./zope.app.wsgi-3.4.0-py2.4.egg ./zope.app.zapi-3.4.0-py2.4.egg ./zope.app.zcmlfiles-3.4.3-py2.4.egg ./zope.app.zopeappgenerations-3.4.0-py2.4.egg ./zope.cachedescriptors-3.4.0-py2.4.egg ./zope.component-3.4.0-py2.4.egg ./zope.configuration-3.4.0-py2.4.egg ./zope.contenttype-3.4.0-py2.4.egg ./zope.copypastemove-3.4.0-py2.4.egg ./zope.datetime-3.4.0-py2.4.egg ./zope.deferredimport-3.4.0-py2.4.egg ./zope.deprecation-3.4.0-py2.4.egg ./zope.dottedname-3.4.2-py2.4.egg ./zope.dublincore-3.4.0-py2.4.egg ./zope.error-3.5.1-py2.4.egg ./zope.event-3.4.0-py2.4.egg ./zope.exceptions-3.4.0-py2.4.egg ./zope.filerepresentation-3.4.0-py2.4.egg ./zope.formlib-3.4.0-py2.4.egg ./zope.hookable-3.4.0-py2.4-macosx-10.3-i386.egg ./zope.i18n-3.4.0-py2.4.egg ./zope.i18nmessageid-3.4.3-py2.4-macosx-10.3-i386.egg ./zope.interface-3.4.0-py2.4-macosx-10.3-i386.egg ./zope.lifecycleevent-3.4.0-py2.4.egg ./zope.location-3.4.0-py2.4.egg ./zope.minmax-1.0-py2.4.egg ./zope.modulealias-3.4.0-py2.4.egg ./zope.pagetemplate-3.4.0-py2.4.egg ./zope.proxy-3.4.0-py2.4-macosx-10.3-i386.egg ./zope.publisher-3.4.1-py2.4.egg ./zope.schema-3.4.0-py2.4.egg ./zope.security-3.4.0-py2.4-macosx-10.3-i386.egg ./zope.session-3.4.1-py2.4.egg ./zope.size-3.4.0-py2.4.egg ./zope.structuredtext-3.4.0-py2.4.egg ./zope.tal-3.4.0-py2.4.egg ./zope.tales-3.4.0-py2.4.egg ./zope.testing-3.5.1-py2.4.egg ./zope.thread-3.4-py2.4.egg ./zope.traversing-3.4.0-py2.4.egg - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev **
Re: [Zope-dev] Re: Egg install bot results
On Wednesday 14 November 2007, Chris McDonough wrote: In the meantime, whose feet would I be shooting if I changed setup.py dependency info so that, e.g. zope.tales (and the like) didn't pull in as many dependencies? Installing zope.tales (as an example, I'm sure its as a result of a transitive dependency lower in the dep chain) right now gets us ~ 75 other distribution, including very unrelated ones: I would welcomes this very much, because I use several packages outside of Zope as well. Now that we have the KGS, I feel much more comfortable with playing with the packages and their dependencies. Note that Marius worked on a nifty little gtk-based tool during the Foilage Sprint that shows you the full dependency tree. http://mg.pov.lt/gtkeggdeps/ Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Egg install bot results
On Nov 14, 2007, at 9:57 AM, Jim Fulton wrote: So what makes this stuff work via buildout is all version-pinning now? They aren't using the cheeseshop. They are using the KGS instead. They are currently also pinned. zope.tal's buildout.cfg contains: [versions] zope.traversing = 3.4.0 zope.app.publisher = 3.4.0 zope.tal shouldn't depend on either of these, so I'd like to remove both the pinning in buildout.cfg and fix the dependency bug that caused them to need to be added in the first place. The actual bug is that zope.tal depends on zope.i18n (which has inappropriate dependencies, but we'll fix that separately). zope.tal only uses the zope.i18n for a single import in dummyengine.py. It imports and uses the zope.i18n.interfaces.ITranslationDomain interface: from zope.i18n.interfaces import ITranslationDomain class DummyTranslationDomain(object): implements(ITranslationDomain) I removed ITranslationDomain implementation from the DummyTranslationDomain, all the tests still pass. I've also removed the pinning in buildout.cfg. I've checked this in. If you do setup.py install within zope.tal now, the only distributions required and downloaded during that process are zope.i18nmessageid and zope.interface. Likewise for buildout. I've also made this module possible to test via setup.py test. - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Egg install bot results
On Wednesday 14 November 2007, Chris McDonough wrote: zope.tal's buildout.cfg contains: [versions] zope.traversing = 3.4.0 zope.app.publisher = 3.4.0 zope.tal shouldn't depend on either of these, so I'd like to remove both the pinning in buildout.cfg and fix the dependency bug that caused them to need to be added in the first place. Just remove the section. I removed them in the packages I saw when I was doing the eggs. Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Egg install bot results
On Wednesday 14 November 2007, Jim Fulton wrote: For a while, those of us working on Zope 3 revelled in refactor mercilessly. This was fine before Zope 3 was widely adopted. We have since leaned the value of mercy. :) LOL! (This should make it into the eternal list of Zope quotes.) Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
