I realize now that I've neglected to give sufficient reasoning for why less
granular packaging would be a good thing.
I've noticed that there's a common theme in Zope development, software, and
packages that I can only describe as "power law development" or "turtles all
way down". It's a bit of an antipattern, unfortunately.
I'll provide an example by way of Zope-3-the-appserver. In an application that
uses Zope-3-the-appserver, many individual subframeworks will be used. For
example, there is a traversal subframework, a security subframework, a
cataloging subframework, and so on.
Each of these subframeworks acts as a logical unit, and through the magic of
component architecture, each can be replaced wholesale by registering some
adapter. However, each of these subframeworks tend to also have settings that
can be configured. For example, individual traversal steps for certain types
objects can be overridden by registering an adapter that *configures* the
subframework. In the case of Zope 3, we have a traversal situation where the
larger traversal subframework can either be replaced wholesale via an adapter
registration or extended piecemeal via other adapter registrations.
The problem is that the mechanism to *replace* the subframework is the same as
the mechanism to *configure* it (both are done via adapter registration,
sometimes even in the same file). This is theoretically fine. But in reality,
it's tremendously hard for someone just walking up to a complex system like
3 to discern adapter registrations that replace subsystems from those which
merely configure subsystems. An inability to discern the difference leads to
situations where people just "don't get the joke" and try to wiggle wires to
configure-to-death a existing subsystem that's clearly suboptimal for their use
case instead of just replacing it wholesale with a much simpler custom policy.
They just don't know it was engineered to be replaced. So they keep adding
configuration code to the existing subframework to handle various 1% edge
Often this code makes the subframework tremendously complex, and the
subframework grows inappropriate dependencies along the way. *Sometimes* the
situation gets so confusing for a new user, they just quit and go use something
This is a pattern that happens over and over again in Zope development. In my
personal opinion, the original error was trying to make the subframework
configurable at all. Instead, the subframework should be replaceable easily,
but it should itself be relatively rigid. At very least, for subframeworks
really do require extra configuration (should be very few), this configuration
should be done via highly specialized ZCML directives (or grokkers), as opposed
to some very general adapter registration that can't be easily discerned from
other adapter registrations by a newbie.
If the subframeworks were more rigid (but replaceable), the *intent* of the
subframework author could be more easily discerned, and fewer people would fall
into the trap of adding more configuration code to a subframework instead of
just replacing it entirely. And fewer people would just walk away in
What does this have to do with packaging? Well, currently, there's a dizzying
number of packages that make up "the ZTK" (nee "Zope 3"). Each of these
packages is a pure peer of all others in a PyPI listing with no real way to get
a sense of their relative importance other than performing a linear audit.
if a user *does* do a linear search of all of them, it's still awful hard to
discern for some new user which ones are "important", and which ones just
to exist by some inequity of history without trying to install it. The user
needs to gain some holistic knowledge of the system in order to discern the
important bits from these historical inequities.
Most new users understandably just walk away from *all* Zope packages before
they gain this knowledge; it's just too hard for them to tell the difference
between the truly important and reusable bits and the stuff that just happens
be packaged up and released but which is useless outside of some highly
context. In effect, we just don't communicate *intent* very effectively in our
current packaging structure.
In my opinion, this is why a lot of Python developers who are otherwise very
smart have given up on trying to use Zope packages. The time required to
out which ones are useful and which ones aren't is just too high. It's way
"easier" for them to write them all off wholesale and just write what they need
from scratch or use somebody else's software. Good developers tend to like
small, useful libraries and frameworks.
We can ameliorate the situation in a few ways:
- We can reduce the number of distributions.
- We can make each current Zope package distribution independently useful.
My suggestion is that we do the former first in order to communicate *current
intent* (the "state of reality right now"). We can do the latter after this in
pieces, hopefully aided by some new developers we've picked up by making it
easier to find useful stuff.
Once we deflate our current set of packages down to a reasonable number, the
packages listed in PyPI will immediately start to reflect "the state of reality
right now". As a result, we'll hopefully be able to get some new blood in the
form of new developers that use the smaller bits outside Zope to help us tease
the truly independent pieces out of the larger pile. If we do this, at no time
after the deflation will PyPI listings ever as badly advertise "the state of
reality" as it is advertised right now, and the community will hopefully again
start to grow.
On 5/11/09 11:11 AM, Martijn Faassen wrote:
> Chris McDonough wrote:
> > Instead, I have argued for promoting packages that have some life of
> > their own (independent of the rest of the pile) into subprojects that
> > have their own release cycles.
> > Then outside projects such as Plone and Grok could depend on
> > independent versions of such packages, giving them slightly more
> > flexibility than requiring a "version of the ZTK".
> We already have that flexibility today. To me, the utility of a release
> of version numbers in the ZTK does not at all exclude the potential to
> evolve the packages to more independent sub-projects.
>> Given that this suggestion has been met with skepticism, let me try another
> I think that's an inaccurate description of the response you got. I'm
> quite positive about trying to give as many packages as possible a life
> of their own. I don't think you got anyone else arguing against this
> point of view.
> I'm also quite positive that some packages are:
> * useful as independently distributed packages
> * only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they
> depend on a significant set of Zope packages.
> I'd like to get out of this paradigm:
> * the Zope packages are independent sub-projects.
> * therefore we cannot distribute a list of versions that work best together.
> And this one:
> * if we distribute a KGS of anything
> * packages in that anything aren't independently reusable automatically
> and should be merged into a ball.
> I'd also like to get out of the following paradigm:
> * the Zope packages are not independently reusable yet
> * therefore we should distribute them all together
> We're in a grey area. Some package are here, some packages are there,
> some are in between. Some packages build on other packages, but have
> clear dependency structures. Some don't have clear dependency
> structures. Some have better documentation and better focus than others.
> If there is to be a merging of code together, then I propose we continue
> the project where the ZMI code is merged into one or a few packages. We
> can also investigate merging 2 or 3 packages together where it seems to
> make sense, or simply moving code between packages (some code needs to
> go down the dependency list, some up).
>> Instead of thinking about it this way, could we think about it as
>> *deflating* the current set of zope.* packages that do not currently have a
>> meaningful life of their own into a single setuptools distribution named
>> This package would include most everything in zope.app*, plus things like
>> zope.server, zope.publisher, and other things that just aren't useful
>> outside of
>> Zope-the-appserver, or which currently just depend on "too much".
> This would make it a lot harder to:
> * clean up dependency relationships with the goal of creating more
> reusable code. We'd all hide them in a sumo ball again.
> * get rid of bits we *don't want anymore*. If I need anything in a sumo
> package I'd need *all* of it.
> * override individual packages with newer versions
> * we've done a lot of refactoring recently trying to separate the UI
> from packages. This is done by creating a *new* package, leaving the old
> package behind. We can do this, test this and release this
> package-by-package now.
>> Over time, we'd tease out the dependencies of packages that live in the ZTK
>> distribution, making them useful outside without any dependency on the ZTK.
>> names of these packages could be arbitrary (they wouldn't need to retain
>> old "zope.*" names). Some backwards dependency shims would be added to the
>> to prevent old imports from breaking, and the ZTK distribution would then
>> have a dependency on the thing that got broken out.
> I don't like the attempt to redefine what the ZTK means to a giant ball
> of Python packages. That's implying that, say, zope.component is *not*
> in the ZTK. That's wrong.
> Why generate a whole lot of work for ourselves getting from where we are
> now to here? We've learned how to work with the current situation in
> 2007 already.
>> I'm thinking that this would simplify things greatly for people who want to
>> consumers of "truly independent" Zope packages. There'd be exactly N of them
>> available for download (where N is much less than 100, more like 20), plus
>> ZTK, which would have the rest of the pile in it over time.
> I don't see why a big package would "simplify things greatly" for you or
> anyone else.
>> If someone wanted
>> to use a forked version of a package that lived in the ZTK distribution,
>> either do so by teasing out the dependencies and making it "truly
>> or they'd just reroll a custom version of the entire ZTK distribution.
> And that's easier than the current situation how? Are you really
> proposing we destroy the dependency information we've already teased out
> and then make people do the work again?
>> Does this make any sense?
> Not a lot in my book.
> I think an important reason why there's so much awareness of dependency
> issues in the Zope world now (and effort spent on it) is precisely
> because we released our separate packages and can see the dependency
> information clearly.
> Zope-Dev maillist - Zope-Dev@zope.org
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope )
Zope-Dev maillist - Zope-Dev@zope.org
** No cross posts or HTML encoding! **
(Related lists -