Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-12 Thread Martijn Faassen
Hey,

On Nov 11, 2007 10:34 PM, Jim Fulton [EMAIL PROTECTED] wrote:

 On Nov 11, 2007, at 2:06 AM, Martijn Faassen wrote:
[snip]

  This breaks a fundamental assumption for releases. When I release
  something, I expect it to work tomorrow, next month, and next year.

 If you want this, then you can't rely on the KGS.  When releasing our
 applications, we don't rely on a KGS. We fix all of the versions
 we're using.  IMO, the KGS shouldn't try to solve this problem.  A
 KGS should be helpful for developers and development frameworks.  A
 KGS will be more useful if the quality remains high.   A KGS is
 similar to a traditional monolithic release.  After all, bug fix Zope
 releases have been known to break applications too.

I got completely confused by your answers you gave previously: you
were talking about feature releases, but of what?

Basically here you say that KGS replaces a monolithic release of Zope
3. I see KGS as useful for the developers of Zope 3 classic. I see KGS
as a useful source of tested lists of versions where they are related
to Zope 3.

  With code, we know that history, and branches, and so on, are
  important. We use Subversion. With KGS we only have an ongoing trunk.

 I'm not sure why you keep saying trunk.  I'm not sure if you are
 being imprecise, or if I'm missing something.
 There's no reason a KGS couldn't be managed with a revision control
 system. That might be a very good idea.

I say this as this is the impression I get from it. Saying that a KGS
could be managed with a revision control system is nice, but can it
now? How complicated would that make it? Does it make sense to
maintain this information externally to the packages?

[snip]
  There are some fundamental problems with external lists or indexes:
 
  * we need to know about the dependency of dependencies, even if we
  never use them directly. Information hiding is broken.

 I'm not sure how this is a problem with version lists (external or
 otherwise) or indexes.

Dependencies of dependencies itself isn't a problem, as this
information is still in packages themselves. The versions of
dependencies isn't, and this is a problem, because I don't *want* to
know about dependencies of dependencies, or their versions. I don't
want to have to care. The packages themselves should know this.

The story for beginners wouldn't be good enough, as they'd need to
know too. With ZCML we're finally resolving this by putting this
dependency structure in ZCML. Not ideal, but at least when you include
package X which needs Y which needs Z, you don't need to manually
include the ZCML of Y and Z anymore. Now with package dependencies and
versioning, I need to make decisions on the versions of Y and Z, while
I just care about using X. I don't want to know this stuff.

A beginner can of course, if he's lucky, interact with an index like
KGS that makes decisions for them. That works until they need a
different version or different package than what is maintained in the
index. In that case, they don't want anything external to make the
decisions.

The basic thing I'd like is to just to ask the package: give me the
versions *you* think you can work with. If I'm tracking this package
in subversion or upgrade to a new release of the package, this list of
best versions might change, too. I don't want to have to know, just
like I don't want to have to know about the implementation details of
a package.

Dependencies and the versions of such are an implementation detail.
One I might on occasion like to override, just like I sometimes need
to override the implementation details of a class by subclassing, but
that should be further along the curve, not immediate. I think it
would be reasonable for packages higher up in the dependency tree to
have the ability to override version decisions made by packages lower
down (as long as there aren't any conflicts within the structure). In
these case these package explicitly decide to take over
responsibility.

  * a single list will never do it. We intend to have many different
  applications that may depend on different versions of packages.
  Grok may need a newer zope.publication than your application does.
  A Grok extension may need an even newer version than Grok does.
  We'll be baking endless amounts of lists this way.

 I think each application will need to come up with a version list for
 each of it's releases.  In development, an application can use an
 index or external version list as a starting point.  For example, I
 see a KGS being useful as a (fairly) stable baseline for
 development.  When an application is ready for release, it should fix
 it's versions.  I've tried to make this easy to do with buildout.
 When you're preparing to make a release, run buildout in verbose mode
 (-v) It will print out the versions it picked in a format that is
 easily turned into a version list.

Sure, I know about all this. I just am saying that this doesn't do
enough. During development I decide I want to 

Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-12 Thread Martijn Faassen
Hey,

On Nov 12, 2007 12:02 AM, Stephan Richter [EMAIL PROTECTED] wrote:
[snip]
 Like Linux distributions, there will be a KGS for every Zope 3
 release. I have already requested a new directory called zope-dev where new
 feature releases can be tested.

Okay, I didn't understand that KGS is replacing the monolithic release
story for Zope 3. That's fine as far as it goes. I was focused on the
ability that eggs give us for packages to move at different speeds of
evolution, and the desire to pick those eggs that we prefer in our
applications. If you don't need that, then KGS is basically Zope 3
release + a few features.

[snip]
  We intend to let packages move at different feature-release speeds,
  and we can't have a KGS for each package.

 You do not need to have a single KGS for every package. But believing that we
 can just randomly make new feature releases that work with the rest of the
 world is naive at best. We have seen already what happens, if everyone uses
 their own set of versions and packages.

Clearly things didn't work in the past. You can't just throw random
versions of eggs together. We couldn't do anything better, as the
information about what worked together was missing. KGS adds that
information back, and that's great. But the information is external to
the actual packages. This has drawbacks. I'm saying that if we add
this information in the packages internally, we'll be better off, as
historical versions and future versions can work. The reuse story is
improved. You can make your own selection of versions and have a
decent chance it will work together.

 A development KGS will be used to test new feature releases.

  What KGS doesn't have is history.

 Yes, it does. Why do you think I manage the controlled-packages.cfg file in
 SVN? And in SVN, I do not create branches and tags without a reason.

Okay, that's a history. It's a history external to the packages, while
the packages have their own history. It's also a global history, while
packages can evolve independently. Development decisions of a
package's development can change dependency information. It therefore
seems natural to me that this information is maintained next to the
package. If it's not, and zope.component starts to rely on a newer
version of zope.interface, I'd need to maintain this centrally with
KGS. We introduce a new monolithic structure where we just removed it.
We add back explicitly what was there implicitly: an SVN trunk of Zope
3 maintaining versions that all work together. That's fine to retain
the features Zope 3 development had, but I thought the point of
splitting Zope 3 up was to be able to forget about the SVN trunk of
Zope 3 and just worry about what's right for zope.component.

[snip]
  With Grok, we use an external versions list. We can use this to solve
  the above problem. We basically take snapshots of what is in KGS. This
  allows us to maintain some history, though it isn't ideal either, as
  it's quite a bit of overhead.

 How is this overhead?

Besides releasing Grok, we also need to maintain snapshots of what is
in KGS, make such changes as are needed, and publish them. Previously
we just released new versions of Grok. That's increased maintenance
and release overhead I'd like to get rid of again.

  If I build an application or framework on top of Grok, I will need to
  maintain yet another external list for the extra packages of this
  application, fixing those versions.

 Why? I don't follow that?

Because these packages may be of different versions that in KGS, or
may not be managed by KGS altogether.

  So, while annoying, that is somewhat manageable. Now imagine I want to
  use a completely separate Python library with my Grok application. This
  python library has dependencies itself again. This means I will need to
  know about versions of those dependencies as well, and fix them into my
  application's list.

 Yes. I see this as an advantage. Version specifications in `setup.py` usually
 contain ranges of allowed versions. What happens if one release in the range
 does not work? Then you make false promises. The only way to avoid this would
 be by specifying all allowed versions exactly, which makes no sense.

That's true. Where I'd like to specify this is as near as possible to
where I make the decision to fix these versions. In case of an
application, that may be the application. Often that's not the case
though: in the case of a library that uses these packages, I'd like it
to be the library, and in case of a framework, I'd like it to be the
framework. When I develop an application at most I'd like to get the
warning: these packages still don't have fixed versions. I'd prefer
that list to be empty.

  There are some fundamental problems with external lists or indexes:
 
  * we need to know about the dependency of dependencies, even if we never
  use them directly. Information hiding is broken.

 This is a requirement not a problem statement. I don't understand Information
 hiding 

Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-12 Thread Jim Fulton


On Nov 11, 2007, at 6:34 PM, Stephan Richter wrote:


On Sunday 11 November 2007, Jim Fulton wrote:

This breaks a fundamental assumption for releases. When I release
something, I expect it to work tomorrow, next month, and next year.


If you want this, then you can't rely on the KGS.  When releasing our
applications, we don't rely on a KGS. We fix all of the versions
we're using.  IMO, the KGS shouldn't try to solve this problem.  A
KGS should be helpful for developers and development frameworks.  A
KGS will be more useful if the quality remains high.   A KGS is
similar to a traditional monolithic release.  After all, bug fix Zope
releases have been known to break applications too.


I really hope you will use the KGS as a starting point somewhen for  
your
internal applications as well. :-) (Note that you can now override  
versions
using the new extends feature that I shamelessly copied from  
buildout.)


And I am not saying this to promote the KGS. I have a concrete  
example.


Probably as part of a project, Benji did some development on  
zope.testbrowser.
He fixed the versions of all dependencies in buildout.cfg. However,  
those
versions were a version sub-graph of a ZC internal dependency graph  
that I do
not have access to. It was also already pretty outdated referring  
to dev

and alpha releases.

So while testbrowser might be working with those dependency  
versions, it might
still be broken for me, because I have a totally different  
dependency graph.
The worst scenario, which luckily has not happened yet, is that we  
fix things

back and forth because of different dependency graphs.

I thus propose that all packages in svn.zope.org should use a KGS  
for testing,
because it is a fully public dependency graph. I am not sure  
whether it
should be the latest stable KGS or the development KGS or whatever.  
Time will

provide an answer.


I think you make a good point.

+1 on using *some* KGS.

Jim

--
Jim Fulton
Zope Corporation


___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-12 Thread Stephan Richter
On Monday 12 November 2007, Jim Fulton wrote:
  I thus propose that all packages in svn.zope.org should use a KGS  
  for testing,
  because it is a fully public dependency graph. I am not sure  
  whether it
  should be the latest stable KGS or the development KGS or whatever.  
  Time will
  provide an answer.

 I think you make a good point.

 +1 on using *some* KGS.

Since we only have the Zope 3.4 KGS now, I think it would be the best one to 
use now. :-)

The easiest way to do this is to add the following line to the buildout 
section of the package's `buildout.cfg` file:

index = http://download.zope.org/zope3.4

(I know you know that Jim; it is for the benefit of people reading this 
mail. ;-)

Regards,
Stephan
-- 
Stephan Richter
CBU Physics  Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-12 Thread Chris Withers

Stephan Richter wrote:
The easiest way to do this is to add the following line to the buildout 
section of the package's `buildout.cfg` file:


index = http://download.zope.org/zope3.4

(I know you know that Jim; it is for the benefit of people reading this 
mail. ;-)


I've been trying to follow this whole thread but it's been pretty high 
volume so apologies if I've missed something...


If I specify index as above, how do I get other packages which may not 
appear in that index?


cheers,

Chris

--
Simplistix - Content Management, Zope  Python Consulting
   - http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-11 Thread Lennart Regebro
On Nov 11, 2007 8:06 AM, Martijn Faassen [EMAIL PROTECTED] wrote:
 I therefore still believe that version dependency information should
 move out of external indexes and into packages.

This is at least the intuitive place for this information. My
application requires Grok 0.11, which requires zope 3.4.0b2 which then
would be a package that doesn't contain any code, just requirements of
eggs that in turn has requirements of their own. I'm not even sure
this *is* different from how the unices does it, but it just seems the
obvious way of doing it. I would be interested in knowing if this has
drawbacks.

-- 
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-11 Thread Wichert Akkerman
Previously Martijn Faassen wrote:
 People have been saying that since Linux distributions use external 
 indexes, we should too, as we are dealing with the same problem as Linux 
 distributions. While the problem is similar, I think the nature of 
 development makes our problems, and therefore our solutions, quite 
 different from the way distributions do it.
 
 How are we different?
 
 We have many, many different small distributions (package + 
 dependencies) that can be combined. We have such a small distribution 
 for each application. We have such a small distribution for each 
 extension. Not just that. We have such a small distribution for each 
 *release* of an application. We have such a small distribution for each 
 *release* of an extension.
 
 I therefore still believe that version dependency information should 
 move out of external indexes and into packages.

Unless I'm missing something that is exactly what Linux distributions
are doing. Each package has its own list of dependencies and conflicts
(just as important). When a package is uploaded to a distribution
archive that information is copied out of the package and included in
the distribution index. That is important since it allows you to grab
the index and calculate the whole dependency graph without having to
download packages. You can know in advance if something is installable
without having to download dozens of pacakges and only then discovering
that it will never work.

Linux package managers can also handle multiple distributions. If you
look at apt for example it can handle as many distributions as you want.
You can set priorities for them at distribution-scale (ie always prefer
packages from distribution X), at release scale (ie always prefer
packages from release Y even if release Z has a newer version) or
package scale (package A has to come from distribution X). This is
extremely common. If you install a Debian or Ubunutu machine you will
always use two distributions: the one for the release, which will never
change, and one with security fixes for just that release. Often you
will also configure distributions with specific backports (needed
because Debian releases are far apart) or for specific products (for
your Enlightement 17 snapshot for example which Debian does not have). 

The terminolpgy is slightly different (archive versus index, package
versus egg, depends versus requires, enlightenment versus grok, etc.)
but the problem is still the same.

Wichert.

-- 
Wichert Akkerman [EMAIL PROTECTED]It is simple to make things.
http://www.wiggy.net/   It is hard to make things simple.
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-11 Thread Jim Fulton


On Nov 11, 2007, at 2:06 AM, Martijn Faassen wrote:


Hi there,

I've been doing some more thinking about external version indexes  
(like Grok's versions.cfg on a URL, and like KGS) and why they  
won't solve all our problems. I have a new way to express it, so  
let me try it out on you all.


What KGS solves is that it allows the ongoing development and  
testing of an integrated Zope 3.


I see it addressing a more general problem of having a known good  
combination of components that work together. There's nothing Zope 3  
specific about this.


That is, there's a Zope 3 'trunk' of versions that keeps being  
updated as there are bugfix releases.


That's not how I see it. As I've said before, I would model this on  
linux distributions, where each feature release has a repository of  
packages for that release, including bug fixes.


I'm not sure what happens as soon as someone wants to make a new  
feature release of any package.


They make a new release.  At some point, someone will make a new KGS  
that incorporates this.


Presumably they end up in KGS too. After all, we won't have a  
single Zope 3.4 and then a single Zope 3.5 for which we can create  
a new KGS.


Why not?  I would expect that there would be Zope 3.4 and Zope 3.5  
KGSs.  There might be additional KGSs that include some of the same  
components. Anyone can assemble a KGS if they think that in doing so,  
they can add value.



We intend to let packages move at different feature-release speeds,  
and we can't have a KGS for each package.


Of course not.

Another problem KGS can solve is to add some release hygiene to the  
cheeseshop: do not remove old releases or overwrite them.


I don't really understand this.  Maybe you mean that a KGS can be a  
better alternative to the cheeseshop. I can certainly see that.


What KGS doesn't have is history. When I release an application or  
framework and I used KGS to make sure that all my versions were  
correct, it will work on the day of release. As soon as enough  
bugfix (or feature) releases make it to KGS, something will  
inevitably break. We've seen innocuous changes breaking code a lot  
of times, so we can't pretend that never happens. It *will* happen.


Yup. Which is why you should record versions you use.

This breaks a fundamental assumption for releases. When I release  
something, I expect it to work tomorrow, next month, and next year.


If you want this, then you can't rely on the KGS.  When releasing our  
applications, we don't rely on a KGS. We fix all of the versions  
we're using.  IMO, the KGS shouldn't try to solve this problem.  A  
KGS should be helpful for developers and development frameworks.  A  
KGS will be more useful if the quality remains high.   A KGS is  
similar to a traditional monolithic release.  After all, bug fix Zope  
releases have been known to break applications too.


With code, we know that history, and branches, and so on, are  
important. We use Subversion. With KGS we only have an ongoing trunk.


I'm not sure why you keep saying trunk.  I'm not sure if you are  
being imprecise, or if I'm missing something.
There's no reason a KGS couldn't be managed with a revision control  
system. That might be a very good idea.


With Grok, we use an external versions list. We can use this to  
solve the above problem. We basically take snapshots of what is in  
KGS. This allows us to maintain some history, though it isn't ideal  
either, as it's quite a bit of overhead.


Yup.  I think both KGSs and version lists are valid approaches.  Each  
has different strengths and weaknesses.


If I build an application or framework on top of Grok, I will need  
to maintain yet another external list for the extra packages of  
this application, fixing those versions. We could probably even use  
the extends feature of buildout to have this list point at Grok's  
list so we have to repeat ourselves less should we want to build  
something on top of *that* application or framework again.


Yup.

So, while annoying, that is somewhat manageable. Now imagine I want  
to use a completely separate Python library with my Grok  
application. This python library has dependencies itself again.  
This means I will need to know about versions of those dependencies  
as well, and fix them into my application's list.


Yes


There are some fundamental problems with external lists or indexes:

* we need to know about the dependency of dependencies, even if we  
never use them directly. Information hiding is broken.


I'm not sure how this is a problem with version lists (external or  
otherwise) or indexes.





* a single list will never do it. We intend to have many different  
applications that may depend on different versions of packages.  
Grok may need a newer zope.publication than your application does.  
A Grok extension may need an even newer version than Grok does.  
We'll be baking endless amounts of lists this way.


I think each application will need to 

Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-11 Thread Stephan Richter
On Sunday 11 November 2007, Lennart Regebro wrote:
 On Nov 11, 2007 8:06 AM, Martijn Faassen [EMAIL PROTECTED] wrote:
  I therefore still believe that version dependency information should
  move out of external indexes and into packages.

 This is at least the intuitive place for this information. My
 application requires Grok 0.11, which requires zope 3.4.0b2 which then
 would be a package that doesn't contain any code, just requirements of
 eggs that in turn has requirements of their own. I'm not even sure
 this *is* different from how the unices does it, but it just seems the
 obvious way of doing it. I would be interested in knowing if this has
 drawbacks.

Meta-eggs are considered a bad idea in the Python world. I originally wanted 
to create a meta-egg, but Jim convinced my to use a different approach; hence 
the index.

Regards,
Stephan
-- 
Stephan Richter
CBU Physics  Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-11 Thread Jim Fulton


On Nov 11, 2007, at 6:11 PM, Stephan Richter wrote:


On Sunday 11 November 2007, Lennart Regebro wrote:
On Nov 11, 2007 8:06 AM, Martijn Faassen [EMAIL PROTECTED]  
wrote:

I therefore still believe that version dependency information should
move out of external indexes and into packages.


This is at least the intuitive place for this information. My
application requires Grok 0.11, which requires zope 3.4.0b2 which  
then
would be a package that doesn't contain any code, just  
requirements of

eggs that in turn has requirements of their own. I'm not even sure
this *is* different from how the unices does it, but it just seems  
the

obvious way of doing it. I would be interested in knowing if this has
drawbacks.


Meta-eggs are considered a bad idea in the Python world. I  
originally wanted
to create a meta-egg, but Jim convinced my to use a different  
approach; hence

the index.


Meta eggs aren't a bad or a good idea by themselves.  They are a good  
solution to some problems and a bad (or less good) solution to  
others.  IMO, meta eggs are a good way to fix versions in  
*applications*.  (I think buildout's version-specification mechanism  
is another good approach, with certain advantages and disadvantages).


I think a package repository, of which a KGS is an example, is a good  
way to provide access to a collection of packages known to work  
together -- especially as it provides a nice way to manage bug  
fixes.  I think Zope 3  is better served by a well-managed  
repository, because Zope 3 is a platform, not an application.   IMO,  
a well-managed KGS (set of KGS releases) will serve the community of  
developers who use Zope better than a rigid version specification.


Jim

--
Jim Fulton
Zope Corporation


___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-11 Thread Stephan Richter
On Sunday 11 November 2007, Jim Fulton wrote:
  This breaks a fundamental assumption for releases. When I release  
  something, I expect it to work tomorrow, next month, and next year.

 If you want this, then you can't rely on the KGS.  When releasing our  
 applications, we don't rely on a KGS. We fix all of the versions  
 we're using.  IMO, the KGS shouldn't try to solve this problem.  A  
 KGS should be helpful for developers and development frameworks.  A  
 KGS will be more useful if the quality remains high.   A KGS is  
 similar to a traditional monolithic release.  After all, bug fix Zope  
 releases have been known to break applications too.

I really hope you will use the KGS as a starting point somewhen for your 
internal applications as well. :-) (Note that you can now override versions 
using the new extends feature that I shamelessly copied from buildout.)

And I am not saying this to promote the KGS. I have a concrete example.

Probably as part of a project, Benji did some development on zope.testbrowser. 
He fixed the versions of all dependencies in buildout.cfg. However, those 
versions were a version sub-graph of a ZC internal dependency graph that I do 
not have access to. It was also already pretty outdated referring to dev 
and alpha releases. 

So while testbrowser might be working with those dependency versions, it might 
still be broken for me, because I have a totally different dependency graph. 
The worst scenario, which luckily has not happened yet, is that we fix things 
back and forth because of different dependency graphs.

I thus propose that all packages in svn.zope.org should use a KGS for testing, 
because it is a fully public dependency graph. I am not sure whether it 
should be the latest stable KGS or the development KGS or whatever. Time will 
provide an answer.

BTW, Benji wanted me to bring this issue up on the mailing list already, so I 
fulfilled my commitment now. :-)

Regards,
Stephan
-- 
Stephan Richter
CBU Physics  Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] why external version indexes don't fulfill all use cases for development

2007-11-10 Thread Martijn Faassen

Hi there,

I've been doing some more thinking about external version indexes (like 
Grok's versions.cfg on a URL, and like KGS) and why they won't solve all 
our problems. I have a new way to express it, so let me try it out on 
you all.


What KGS solves is that it allows the ongoing development and testing of 
an integrated Zope 3.  That is, there's a Zope 3 'trunk' of versions 
that keeps being updated as there are bugfix releases. I'm not sure what 
happens as soon as someone wants to make a new feature release of any 
package. Presumably they end up in KGS too. After all, we won't have a 
single Zope 3.4 and then a single Zope 3.5 for which we can create a new 
KGS. We intend to let packages move at different feature-release speeds, 
and we can't have a KGS for each package.


Another problem KGS can solve is to add some release hygiene to the 
cheeseshop: do not remove old releases or overwrite them.


What KGS doesn't have is history. When I release an application or 
framework and I used KGS to make sure that all my versions were correct, 
it will work on the day of release. As soon as enough bugfix (or 
feature) releases make it to KGS, something will inevitably break. We've 
seen innocuous changes breaking code a lot of times, so we can't pretend 
that never happens. It *will* happen.


This breaks a fundamental assumption for releases. When I release 
something, I expect it to work tomorrow, next month, and next year.


With code, we know that history, and branches, and so on, are important. 
We use Subversion. With KGS we only have an ongoing trunk.


With Grok, we use an external versions list. We can use this to solve 
the above problem. We basically take snapshots of what is in KGS. This 
allows us to maintain some history, though it isn't ideal either, as 
it's quite a bit of overhead.


If I build an application or framework on top of Grok, I will need to 
maintain yet another external list for the extra packages of this 
application, fixing those versions. We could probably even use the 
extends feature of buildout to have this list point at Grok's list so we 
have to repeat ourselves less should we want to build something on top 
of *that* application or framework again.


So, while annoying, that is somewhat manageable. Now imagine I want to 
use a completely separate Python library with my Grok application. This 
python library has dependencies itself again. This means I will need to 
know about versions of those dependencies as well, and fix them into my 
application's list.


There are some fundamental problems with external lists or indexes:

* we need to know about the dependency of dependencies, even if we never 
use them directly. Information hiding is broken.


* a single list will never do it. We intend to have many different 
applications that may depend on different versions of packages. Grok may 
need a newer zope.publication than your application does. A Grok 
extension may need an even newer version than Grok does. We'll be baking 
endless amounts of lists this way.


If this information is inside the packages itself, the history will be 
automatically maintained with Subversion and existing releases. History 
therefore works: if I install Grok 0.11, I would get all dependencies of 
Grok 0.11 automatically without having to worry about external indexes.


Information hiding works: if I use foo 1.3 and foo 1.3 knows it needs 
bar 1.7, it'll simply get that and I don't have to know about it. I 
don't even need to worry about the *existence* of bar.


People have been saying that since Linux distributions use external 
indexes, we should too, as we are dealing with the same problem as Linux 
distributions. While the problem is similar, I think the nature of 
development makes our problems, and therefore our solutions, quite 
different from the way distributions do it.


How are we different?

We have many, many different small distributions (package + 
dependencies) that can be combined. We have such a small distribution 
for each application. We have such a small distribution for each 
extension. Not just that. We have such a small distribution for each 
*release* of an application. We have such a small distribution for each 
*release* of an extension.


I therefore still believe that version dependency information should 
move out of external indexes and into packages.


See also my earlier discussion of these problems and possible solutions:

http://faassen.n--tree.net/blog/view/weblog/2007/09/26/0

Regards,

Martijn

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )