Re: [Chicken-hackers] Distributed egg repo proposal
On 17 March 2011 12:26, Peter Bex peter@xs4all.nl wrote: I haven't had time to digest this fully, but here are some initial remarks: Your proposal ignores one of the main design goals of the distributed egg proposal; it is not backwards compatible; from what I understood, you want to modify chicken-install. Correct. The other remark I have is that it would require substantial rethinking and recoding of our tools, including making a full-fledged package manager from chicken-install complete with user-overlaid local patches. Felix has said before he doesn't want that, and I support him in that; OSes have package managers, and chicken-install should stay simple. With a one-stop egg install tool, which also manages dependencies (as chicken-install currently does), there needs to be smarts somewhere in the system. At what point does one draw the line and call it an overblown package manager, I'm not sure :) At its essence, this portage-like system: * keeps the maintained tree simple. You can see everything at a glance, and sources are kept out. * keeps the load on egg authors as low as possible (you can keep your existing dev workflow, and submit new versions of eggs just by updating a single symbol in one file - and pushing that to the tree maintainers) * pushes the complexity into a single tool. This next-gen chicken-install can start out simple, and functionality can be added over time. * makes it a simple one-liner if you want access to the sources (with full history). * we don't have to deal with the complexities of integrating outside repo's with our repo, etc. Finally, I think your approach also requires people to submit their finished eggs somewhere as a tarball once they've tagged a release? That also ignores the design goal no extra hassle. Not exactly (see (5 6) Making it easier for egg authors from the previous mail). (get-files) and (get-docs) can use any or all of the following: (unpack some_archive 'type_of_archive) (fetch-from-git 'tag) (fetch-from-mercurial 'tag) The egg author must: 1. have his code available somewhere, either within a repository, or as an archive which is publically accessible. 2. give us instructions on: * how to compile (setup ...) * how this egg fits into the tree of eggs (meta ...) * how to fetch the code (get-files), (get-docs) No matter how you look at it, this is the bare minimum for an egg author to do - no matter how we install eggs. A portage-like system doesn't add to this bare minimum - it just arranges things differently (hopefully, in an easier to use manner than we have now). -- Regards, Imran Rafique ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On 17 March 2011 17:56, Felix fe...@call-with-current-continuation.org wrote: Please take a moment to appreciate the excessively careful tone of this reply, something that is quite untypical for me. :-) I'm taking the liberty of responding to each issue in a seperate mail. I must cut down on these War and Peace novels I'm spamming this thread with :) (and apologies if I'm taking this thread too far from the original topic) -- Regards, Imran Rafique ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Imran said: Why a local tree? Because this makes it easier to overlay a personal tree on top. In ~/.chicken-install, I can define a local overlay to be applied on top of the official tree. I can pop my new egg recipes in this local tree, and try them out, before submitting them upstream. Felix said: When mixing trees that way, how do you ensure the combination of them is consistent? The global tree. I can't write to this. + networking/ + zmq/ + recipe-zmq-1.0 + recipe-zmq-2.0 + patches/ I want to add some of my own changes to zmq, which haven't been accepted upstream by the zmq egg authors yet - but I need them for my locally deployed chicken scheme codebase. I want to use the benefits of the egg infrastructure for deploying this modified zmq lib locally (possibly across multiple machines) The local tree is empty, and I just add the following: + networking/ + zmq/ + recipe-zmq-2.0.1 + patches/ + my-changes-1 + my-changes-2 If I run `chicken-install zmq-1.0` or `chicken-install zmq-2.0`, then chicken-install finds the recipes for these in the global tree, so it ignores the local tree *entirely*. If I run `chicken-install zmq-2.0.1`, then chicken-install doesn't find any matching recipes in the global tree, so it looks in the local tree before throwing an error. It finds a recipe for zmq-2.0.1, so it procedes to use the local tree. For patches, it looks in the local_tree/networking/zmq/patches , and then also in global_tree/networking/zmq/patches (so you can avoid needlessly duplicating some patches from the global tree that you might still be using). Or we can just use the patches/ dir from the tree we're using, if that makes things clearer. -- Regards, Imran Rafique ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Imran said:
chicken-install first looks in *local-pkgs* and then *pkgs* to see if
my-wonderful-egg-2.0.tgz exists. If it doesn't, then chicken-install
looks upstream (from upstream-pkgs). If chicken-install had to fetch
something from upstream-pkgs, then this is automatically copied to
*local-pkgs*. This gives rise to a number of possibilities:
Felix said:
How can we be sure the system where the egg is to be installed
has tar/gunzip? (a pure mingw32 build doesn't have this).
Good point.
('ci' = 'chicken-install')
ci is used in 2 different contexts:
1. To populate a tree. Use `ci --fetch-only` on a range of eggs, to
grab the upstream files, and populate *local-pkgs*. This would be how
the global tree maintainers ensure that *pkg* is populated (as
described in my first mail).
In this context, our ci tool is dependent upon certain local
infrastructure being present. If we want to be able to pull from an
upstream git repo (fetch-from-git), then we need git installed.
Similarly with other VCS's we're supporting pulling from. Same with
tar, and any other archive formats we want to allow egg authors to
submit their work in.
2. To install eggs. By regular users, on their machines, on different
architectures. Where we're making no assumptions on whats available
locally.
Well, in this instance, no matter what instructions the recipe file
contains, we're assuming that the global *pkgs* already has the stuff.
ci looks there first. (no outside deps once *pkgs* has been seeded, as
explained in the first mail)
So, no matter what type of archive/source tree we obtained in
context (1), ci is free to write the cached copy of that content (in
*local-pkgs*) in our own format. And in this context (2), thats the
format ci is going to use.
The only assumption we're going to make about the host system is
that we have the base chicken-scheme distribution already installed
and available. So our archive format is something which we can easily
unpack just using the tools the base distribution provides.
--
Regards,
Imran Rafique
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Imran said: * we can have recipe-my-wonderful-egg-latest, where chicken-install grabs the sources from the upstream repos HEAD. * we can have recipe-my-wonderful-egg-felix-2.0, pointing to Felix's forked version of my egg (different info for upstream) Felix said: Forking eggs brings up the consistency problem again. I don't want to fork, there should be no forks, I'd say. I disagree. We may be talking about different things here. Lets assume a new egg, dynamite, is added to the tree. Becuase of the new freedoms this system allows, dynamite's author keeps his repo and workflow on github, and just submits a recipe which specifies the repo details, and tag for each version of dynamite he wants to release. Dynamite is popular, and gets repeatedly forked on github. Many of these forks result in patches flowing back to dynamites author. But there are 2 other forks (by Peter and Felix) where a fundamental architectural difference exists in how some backend stuff is done. To any user of dynamite, any of the 3 are perfect drop-in replacements for each other. The exact same API, etc. The only difference is that they have different performance characteristics in certain situations. Both Peter and Felix endeavour to remain API compatible with the main dynamite. How do you handle this in the official egg tree? We could have 3 seperate eggs, dynamite, dynamite-peter and dynamite-felix. If there were any API differences at all, that would be the cleanest way to handle it. But in this instance, one could consider that its really 3 different flavours of the same egg. So, perhaps this might be a better way to do it? dynamite/ recipe-dynamite-1.0 recipe-dynamite-1.0-felix recipe-dynamite-1.0-peter If dynamite is getting auto-installed as part of a dependency, by ci, then it installs dynamite-1.0. But if *I* am explcitly deciding which dynamite I want, I can choose: chicken-install dynamite chicken-install dynamite-1.0-felix chicken-install dynamite-1.0-peter All 3 recipes provide the same symbol to the dependency tree (dynamite). This may never come up, or we may prefer to have 3 eggs all doing exactly the same thing. But the point is, its possible (as opposed to impossible, with the current setup) and easily manageable. -- Regards, Imran Rafique ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On Fri, Mar 18, 2011 at 08:59:10AM +, Imran Rafique wrote: Felix said: When mixing trees that way, how do you ensure the combination of them is consistent? The global tree. I can't write to this. + networking/ + zmq/ + recipe-zmq-1.0 + recipe-zmq-2.0 + patches/ I want to add some of my own changes to zmq, which haven't been accepted upstream by the zmq egg authors yet - but I need them for my locally deployed chicken scheme codebase. I want to use the benefits of the egg infrastructure for deploying this modified zmq lib locally (possibly across multiple machines) I don't see why chicken-install needs to be bloated with all this stuff. Why not maintain a mercurial patch queue or use quilt. You could also just have a local clone of a repo with your own changes? Adding all this stuff to chicken-install needlessly complicates an intentionally simple tool. Cheers, Peter -- http://sjamaan.ath.cx -- The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music. -- Donald Knuth ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On Fri, Mar 18, 2011 at 09:00:20AM +, Imran Rafique wrote: Dynamite is popular, and gets repeatedly forked on github. Forking's bad, mmmkay? How do you handle this in the official egg tree? We could have 3 seperate eggs, dynamite, dynamite-peter and dynamite-felix. If there were any API differences at all, that would be the cleanest way to handle it. But in this instance, one could consider that its really 3 different flavours of the same egg. To me that seems to be a completely hypothetical situation and unlikely to happen in the real world. Let's keep the discussion grounded in reality, okay? Cheers, Peter -- http://sjamaan.ath.cx -- The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music. -- Donald Knuth ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On 18 March 2011 09:15, Peter Bex peter@xs4all.nl wrote: On Fri, Mar 18, 2011 at 08:59:10AM +, Imran Rafique wrote: Felix said: When mixing trees that way, how do you ensure the combination of them is consistent? The global tree. I can't write to this. + networking/ + zmq/ + recipe-zmq-1.0 + recipe-zmq-2.0 + patches/ I want to add some of my own changes to zmq, which haven't been accepted upstream by the zmq egg authors yet - but I need them for my locally deployed chicken scheme codebase. I want to use the benefits of the egg infrastructure for deploying this modified zmq lib locally (possibly across multiple machines) I don't see why chicken-install needs to be bloated with all this stuff. Why not maintain a mercurial patch queue or use quilt. You could also just have a local clone of a repo with your own changes? Right. Till you have a situation where you've added customisations to a dozen eggs (perhaps your org is deploying scheme on some non-standard environment, necessitating these changes), and you need to manage deploying all these customised eggs to multiple hosts. And you end up wishing you could use the regular chicken egg installation processes ... I don't agree with the bloat assesment. If ci can read a global tree, it can read a local one which has exactly the same format. If it only looks in the local tree if it can't find the requested egg in the global tree, then it doesn't get in the way of anything - until you *need* this functionality and then create your local tree. Adding all this stuff to chicken-install needlessly complicates an intentionally simple tool. Going back to your previous comment regarding not wanting to touch chicken-install. If that is really the case, then everything I've said here is moot anyway. I'm just wasting everyone's bandwidth. -- Regards, Imran Rafique ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On Fri, Mar 18, 2011 at 09:53:19AM +, Imran Rafique wrote: I don't see why chicken-install needs to be bloated with all this stuff. Why not maintain a mercurial patch queue or use quilt. You could also just have a local clone of a repo with your own changes? Right. Till you have a situation where you've added customisations to a dozen eggs (perhaps your org is deploying scheme on some non-standard environment, necessitating these changes), and you need to manage deploying all these customised eggs to multiple hosts. First, that way of working creates more problems than it's worth; this will make it a pain in the ass to maintain. But let's say you do this. Chicken-install already supports this; just point it at a directory to install. You can even run your own in-house henrietta from which you distribute your modified egg tree. Chicken-install is *not* the place to fix this. Adding all this stuff to chicken-install needlessly complicates an intentionally simple tool. Going back to your previous comment regarding not wanting to touch chicken-install. If that is really the case, then everything I've said here is moot anyway. It's not cast in stone, but I don't think anyone's willing to take the effort to add this to chicken-install, or maintain the resulting pile of code when you would. If you're willing to implement this *and* take up maintenance of chicken-install, I suppose there's no problem. Cheers, Peter -- http://sjamaan.ath.cx -- The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music. -- Donald Knuth ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On 18 March 2011 09:20, Peter Bex peter@xs4all.nl wrote: On Fri, Mar 18, 2011 at 09:00:20AM +, Imran Rafique wrote: Dynamite is popular, and gets repeatedly forked on github. Forking's bad, mmmkay? Are you lecturing me, or the denizens of github? -- Regards, Imran Rafique ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On Fri, Mar 18, 2011 at 10:02:17AM +, Imran Rafique wrote: On 18 March 2011 09:20, Peter Bex peter@xs4all.nl wrote: On Fri, Mar 18, 2011 at 09:00:20AM +, Imran Rafique wrote: Dynamite is popular, and gets repeatedly forked on github. Forking's bad, mmmkay? Are you lecturing me, or the denizens of github? It's long been general wisdom that forking should be avoided at all costs, not because of technical reasons but because of social reasons. So yes, I think the way github's promotes rampant forking is bad. See also http://sheddingbikes.com/posts/1299555462.html and http://producingoss.com/en/forks.html If a fork is made that's okay, but *only* as long as that fork is short-lived and integrated back into the mainline tree. And in the long run that means you probably want to wait for that event before starting to use that code, generally. If not, you will only have to use the forked repo temporarily. Cheers, Peter -- http://sjamaan.ath.cx -- The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music. -- Donald Knuth ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On 18 March 2011 10:08, Peter Bex peter@xs4all.nl wrote: On Fri, Mar 18, 2011 at 10:02:17AM +, Imran Rafique wrote: On 18 March 2011 09:20, Peter Bex peter@xs4all.nl wrote: On Fri, Mar 18, 2011 at 09:00:20AM +, Imran Rafique wrote: Dynamite is popular, and gets repeatedly forked on github. Forking's bad, mmmkay? Are you lecturing me, or the denizens of github? It's long been general wisdom that forking should be avoided at all costs, not because of technical reasons but because of social reasons. So yes, I think the way github's promotes rampant forking is bad. See also http://sheddingbikes.com/posts/1299555462.html and http://producingoss.com/en/forks.html If a fork is made that's okay, but *only* as long as that fork is short-lived and integrated back into the mainline tree. And in the long run that means you probably want to wait for that event before starting to use that code, generally. If not, you will only have to use the forked repo temporarily. Well, its all down to the social aspects of open-source, as you pointed out. The key reason why I thought of translating something portage-like to chicken eggs is I really think it offers maximum flexibility (with not too much overhead in tooling - the original portage tooling was a *lot* simpler than it is today). These points regarding forking were just illustrations of just how much additional flexibility it could give, if you really wanted to push it but still stay within the confines of (as Felix called it) The System :) They're not the main selling points of a portage-like architecture, which really are: * clean seperation between upstream sources, and the framework needed to organise and distribute them * but still have easy access to upstream repos (the tool just makes it easy for you to clone, provided you have the necessary local VCS tools to do so) * require as little bookkeeping work as possible to submit eggs * ensure no broken deps, once *pkgs* is seeded. * make it easy to fix broken eggs at the global tree level, if we can't wait for the fixes to be propogated upstream and then pushed back in the normal manner. But now I'm just repeating myself again, and clogging up this thread from perhaps more closer debate on your specific proposal :) (and btw, the generally accepted scenario where the main chicken devs patch a number of eggs which are trivially broken by a new chicken update - thats also a fork from upstream) ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Peter Bex scripsit: It's long been general wisdom that forking should be avoided at all costs, not because of technical reasons but because of social reasons. Well, not at *all* costs. Some forks are essential, like gcc/egcs and XFree86/X.org. The list of stable forks is very small, maybe a dozen: FreeBSD/NetBSD, NetBSD/OpenBSD, byacc/bison, Emacs/XEmacs are the ones that come to mind right away. -- Si hoc legere scis, nimium eruditionis habes. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Its been interesting reading through this thread. I'd like to suggest some amendments to Peter's proposals, addressing some of the concerns highlighted. Specifically: Felix: WE DON'T WANT BROKEN LINKS. As you correctly point out, a non-existent deep dependency will be horrible indeed (catastrophic, actually). How to avoid broken links or missing eggs without a centralized repository I can't say. = Back to basics = In essence, we're trying to provide a single blessed tree of eggs. In itself, this doesn't have to be in a VCS (although thats a useful tool to maintain the tree in). All thats relevant is that, at the end, we have a central, blessed tree which allows us to install eggs and their respective docs - without potentially breaking because some upstream repo's are offline. (By upstream, I mean egg author's own repos, where the main development of their egg occurs) Requirements: 1. if you just want to install an egg (any version), you only need the official blessed tree. 2. if you want to track development, where do I go so I can see the full version history? 3. if you want to add some contributions, where do I go? 4. if some emergency bugfixes are needed, where do they go? 5. how easily can I submit an egg for addition 6. how easily can I update an existing egg 2 and 3 are effectively the same, solved by: * a pointer to a repo somewhere * a convenient one-liner to check out a working-copy or clone from that repo * a link to someone who owns that repo (hey, I've got a great patch. Can I have commit access / email it to you / chat about it?) 3 and 4 are *NOT* the same. Consider what happens when we have a new chicken release, which might (hypothetically) break a dozen eggs (messy in real life too!). Felix (or other dev) knows what the breakages are, and knows what simple fixes are required, and where (salmonella to the rescue). With our current setup, he can just commit the fixes directly in the svn egg repo. That solves the immediate problem. But he has to make sure that those fixes also travel upstream to the main dev repo for each egg (or else a new egg release will potentially re-introduce the same problem, if those devs were still working with the previous chicken version). This is sort of manageable now, where everyone knows everybody. But we're talking about building a system which can handle many many more eggs from many many more contributors. This just won't scale. We have the worst of both worlds right now. We're sort-of forcing all egg contributors to use the existing svn repo - with all the hassle that entails (getting commit access, importing sources, etc), while also having to deal with more than 1 place for a bugfix to be pushed to (so having that central repository doesn't really help, in the long run). = An amendment to Peter's proposal = I've taken the gentoo portage system (in its simplest form), and tried to see how a similar concept might translate to chicken's eggs. Explaining in a little detail, for those who may be unfamiliar with portage ... == The tree == A blessed tree of 'recipes', subdivided into categories (as we have now), dictates how every egg is installed. Our install tool, chicken-install, operates on a LOCAL copy of this tree, which is first fetched (from the offical egg tree repo at call-cc.org) by: chicken-install --update-tree Every time chicken-install is run, it should check if the local tree is up to date, and warn if not. The tree looks like this: eggs/ + networking/ + my-wonderful-egg/ + recipe-my-wonderful-egg-1.0 + recipe-my-wonderful-egg-2.0 + patches/ + fix-for-chicken-4.9.patch We have 1 recipe FOR EACH version of the egg. This tree, once obtained - works 'as is' for all versions of all eggs. You don't need to rewind your local working copy of this tree just to install an older version of an egg. You always want the latest version of the official tree. So we have 1 recipe *for each* version of the egg, containing setup meta s-expr's, and additional info (see below). Why a local tree? Because this makes it easier to overlay a personal tree on top. In ~/.chicken-install, I can define a local overlay to be applied on top of the official tree. I can pop my new egg recipes in this local tree, and try them out, before submitting them upstream. Also, in certain circumstances, like re-installing an egg where all the dependencies are now available locally - you don't need network access. So why enforce that need? As you read on, please remember that at no time are we importing sources (or tarballs) from upstream into this tree. Thats not necessary in order to avoid broken dependencies. = (1) Installing an egg = chicken-install takes the recipe, and from that knows where to get the files for that version. Wait a minute. Outside dependency? Well, no. We have 2 stores (lets say *pkgs* and *local-pkgs*). *pkgs* points to somewhere on call-cc.org, and *local-pkgs* (default #f) can be redefined to
Re: [Chicken-hackers] Distributed egg repo proposal
On Thu, Mar 17, 2011 at 12:16:19PM +, Imran Rafique wrote: Its been interesting reading through this thread. I'd like to suggest some amendments to Peter's proposals, addressing some of the concerns highlighted. I haven't had time to digest this fully, but here are some initial remarks: Your proposal ignores one of the main design goals of the distributed egg proposal; it is not backwards compatible; from what I understood, you want to modify chicken-install. The other remark I have is that it would require substantial rethinking and recoding of our tools, including making a full-fledged package manager from chicken-install complete with user-overlaid local patches. Felix has said before he doesn't want that, and I support him in that; OSes have package managers, and chicken-install should stay simple. Finally, I think your approach also requires people to submit their finished eggs somewhere as a tarball once they've tagged a release? That also ignores the design goal no extra hassle. Cheers, Peter -- http://sjamaan.ath.cx -- The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music. -- Donald Knuth ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Hi, Imran! Many thanks for this detailed information. I'm personally not overly familiar with portage, but what you suggest is very interesting and sounds like an approach that could be made to work. Please allow me to bring up the problems I see with this approach (or with every future proposal). It's not intended to blindly bash your suggestions, but to simply show up things that we should take into account when thinking about... THE SYSTEM. In essence, we're trying to provide a single blessed tree of eggs. In itself, this doesn't have to be in a VCS (although thats a useful tool to maintain the tree in). All thats relevant is that, at the end, we have a central, blessed tree which allows us to install eggs and their respective docs - without potentially breaking because some upstream repo's are offline. (constantly #t) We have the worst of both worlds right now. We're sort-of forcing all egg contributors to use the existing svn repo - with all the hassle that entails (getting commit access, importing sources, etc), while also having to deal with more than 1 place for a bugfix to be pushed to (so having that central repository doesn't really help, in the long run). It's actually working quite fine. That patches will not flow quickly enough to upstream is indeed right, but I don't see that as a critical point. It is more important that the egg tree is in a working state: there are many more users than contributors and it has to scale into that direction as well. Communicating patches upstream is more a problem of the responsiveness of the original maintainer, not so much of the system that stores our eggs, IMHO. Why a local tree? Because this makes it easier to overlay a personal tree on top. In ~/.chicken-install, I can define a local overlay to be applied on top of the official tree. I can pop my new egg recipes in this local tree, and try them out, before submitting them upstream. When mixing trees that way, how do you ensure the combination of them is consistent? chicken-install first looks in *local-pkgs* and then *pkgs* to see if my-wonderful-egg-2.0.tgz exists. If it doesn't, then chicken-install looks upstream (from upstream-pkgs). If chicken-install had to fetch something from upstream-pkgs, then this is automatically copied to *local-pkgs*. This gives rise to a number of possibilities: How can we be sure the system where the egg is to be installed has tar/gunzip? (a pure mingw32 build doesn't have this). * we can have recipe-my-wonderful-egg-latest, where chicken-install grabs the sources from the upstream repos HEAD. * we can have recipe-my-wonderful-egg-felix-2.0, pointing to Felix's forked version of my egg (different info for upstream) Forking eggs brings up the consistency problem again. I don't want to fork, there should be no forks, I'd say. Now its much easier to track the real source history of any egg: chicken-install --upstream-checkout So, without going through the hassle of forcing egg authors to push their sources to an egg svn repo (and either abandoning their existing VCS, or keeping their personal VCS in sync with the egg svn), we still have easy immediate access to the entire history of their egg's source code. How would that be working? Would this require the VCS that holds the code to be installed? Felix (who has unfettered write access to the entire tree) adds a patch to the patches/ dir, creates a new 2.0-r1 (or 2.0.1, as long as we're consistant) recipe copied from 2.0, and adds the following: (get-files ... (apply-patches fix-for-chicken-4.9.patch)) chicken-install will look under patches/ for fix-for-chicken-4.9.patch, and apply it to the sources it already got. That will require a patch tool on the client system, if this is done by chicken-install. We can get far beyond simple patches here, and maybe add other rules, like: * (apply-rename ) which will do a search replace from one deprecated function name to a new one. I'm not sure this can be reliably implemented, in the face of syntax and modules. A final note: Such a system as you describe (or the many possible variations of it) will be of considerably complexity and will need a long testing phase. This is not a point of criticism (any reasonably smart system will be complex anyway), but we should be aware of the fact that it will require manpower, and developers who are willing to put a significant time into making this work - on all platforms. Do we have the resources for that? Thanks again for your input. Please take a moment to appreciate the excessively careful tone of this reply, something that is quite untypical for me. :-) cheers, felix ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Felix scripsit: I like the idea of tgz-submission (did John suggest it? I can't remember, it's already 60 seconds later). How about a central store that accepts tgz's and external clients, that keep their own infrastructure (and DVCS's), who regularly pull repos from the actual contributors and have a simple tool to upload the egg? That way we could decouple the DVCS-Egg process from the infrastructure that stores and serves the egg. The bad bit, as I was saying on #chicken, is that making and publishing tarballs automatically isn't very easy unless you run a server. For example, if you host on Google Code, there's no way to persuade it to construct a tarball for the central site to pull. So either I have to generate a tarball by hand and upload it somewhere else (my web site, say), or I have to run my own server, which I'm not set up to do. -- John Cowan co...@ccil.org http://www.ccil.org/~cowan Today an interactive brochure website, tomorrow a global content management system that leverages collective synergy to drive outside of the box thinking and formulate key objectives into a win-win game plan with a quality-driven approach that focuses on empowering key players to drive-up their core competencies and increase expectations with an all-around initiative to drive up the bottom-line. --Alex Papadimoulis ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On Mar 14, 2011, at 10:12 PM, John Gabriele wrote: 2. *Documentation*. Python's [Cheeseshop](http://cheeseshop.python.org) has a neat feature where, if your project has html documentation, you can (manually) upload it directly to http://packages.python.org/project-name . This way, contributors aren't forced to use any particular source format for their docs -- as long as they can generate html from it before uploading [^1]. Availability of *all* documentation via the wiki, chicken-doc and chickadee is a huge selling point and is important to Chicken. Python has docstrings, Chicken doesn't. Jim ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On Mar 15, 2011, at 12:55 AM, John Gabriele wrote: My point was that, if you tell prospective egg contributors that they can optionally supply their own html docs, they can use whatever tools they want to generate that html (even write it by-hand, if they like). [...] you could offer them the option of just providing their own raw html and you could install it and host it for them at something like http://eggdocs.call-cc.org/package-name. Sounds like a huge *dis*advantage to me. Jim ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On Tue, Mar 15, 2011 at 02:19:20AM -0400, John Cowan wrote: Felix scripsit: I like the idea of tgz-submission (did John suggest it? I can't remember, it's already 60 seconds later). How about a central store that accepts tgz's and external clients, that keep their own infrastructure (and DVCS's), who regularly pull repos from the actual contributors and have a simple tool to upload the egg? That way we could decouple the DVCS-Egg process from the infrastructure that stores and serves the egg. The bad bit, as I was saying on #chicken, is that making and publishing tarballs automatically isn't very easy unless you run a server. Did you even read the proposal? There's a section that mentions how to work around that for subversion and other tools that don't offer tarballs but do offer a way to obtain the raw files. Google code offers raw access to files. For example, if you host on Google Code, there's no way to persuade it to construct a tarball for the central site to pull. So either I have to generate a tarball by hand and upload it somewhere else (my web site, say), or I have to run my own server, which I'm not set up to do. I agree that manually uploading tarballs would be a nuisance. One of the design goals is *no extra work* for egg authors (I suppose this other guy didn't read the proposal either; apparently we're just bikeshedding here) and manual egg submission is extra work. We used to work like this about 6 years ago, before we had the svn tree. It was annoying. Also, for tarball submission we'd also need to write extra code, unless we *really* want to revert to what we had 6 years ago, where people were expected to mail their tarballs to felix directly for him to publish on the web. Cheers, Peter -- http://sjamaan.ath.cx -- The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music. -- Donald Knuth ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Hi John,
On Mon, 14 Mar 2011 23:12:06 -0400 John Gabriele jmg3...@gmail.com wrote:
1. *Lower Barriers to Entry*. My guess is that the absolute easiest
way for potential contributors to contribute a package is to simply
ask them to upload a tar.gz archive of their project. Supply them with
a handful of requirements, such as:
* name format for the tar.gz file {like so},
* inside the unpacked project dir, its directory structure
should look {like this},
* it should contain a package meta-data file named
{such-and-such}, and the file should contain {the following sorts of
data}.
Then the contributor would not even have to know what sort of
archiving or version control is done on the Chicken/Egg side of
things. And they could use whatever version control they like.
That's not much different from the approach some egg developers use
nowadays. The difference is that, instead of submiting a tarball, they
add their new releases to svn.
Best wishes.
Mario
--
http://parenteses.org/mario
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
From: John Cowan co...@mercury.ccil.org Subject: Re: [Chicken-hackers] Distributed egg repo proposal Date: Tue, 15 Mar 2011 02:19:20 -0400 Felix scripsit: I like the idea of tgz-submission (did John suggest it? I can't remember, it's already 60 seconds later). How about a central store that accepts tgz's and external clients, that keep their own infrastructure (and DVCS's), who regularly pull repos from the actual contributors and have a simple tool to upload the egg? That way we could decouple the DVCS-Egg process from the infrastructure that stores and serves the egg. The bad bit, as I was saying on #chicken, is that making and publishing tarballs automatically isn't very easy unless you run a server. For example, if you host on Google Code, there's no way to persuade it to construct a tarball for the central site to pull. So either I have to generate a tarball by hand and upload it somewhere else (my web site, say), or I have to run my own server, which I'm not set up to do. I don't know what you said on #chicken, but my point was that the original maintainer is not the one who pushes the tarball. It could be done by others, possibly automatically. The one who publishes the original code is not involved in the whole process other that putting up his/her code on the internet. Other persons (with access to the appropriate dvcs) could run a semiautomatic task to pull releases and submit tarballs to us. cheers, felix ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
* John Cowan co...@mercury.ccil.org [110315 00:45]: Christian Kellermann scripsit: - Automatically *mirror* the contributed archives on call-cc.org, maybe after the code has passed N salmonella rounds of master and experimental branches. What is an archive in this context? Is this a .tar.gz of a whole remote repository in some random format (git, hg, bzr, remote svn)? This is the issue that bothers me. Using one DVCS would be great. Using *every* DVCS at the whim of the egg maintainer is an invitation to chaos. And the magic script will in the nature of things only support a subset of all the CVSes out there. I was indeed talking about the whole repository. And no this does not have to mean every obscure DVCS that's alive on this planet. To lower the barrier, we could agree on a set of supported systems. I am actually against release tarballs, as they throw away the history. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On Tue, Mar 15, 2011 at 3:06 AM, Jim Ursetto zbignie...@gmail.com wrote:
On Mar 14, 2011, at 10:12 PM, John Gabriele wrote:
2. *Documentation*. Python's
[Cheeseshop](http://cheeseshop.python.org) has a neat feature where,
if your project has html documentation, you can (manually) upload it
directly to http://packages.python.org/project-name . {snip}
{snip}
Python has docstrings, Chicken doesn't.
Hi Jim,
The Python package docs at packages.python.org/package-name (and not
many packages are actually making use of the doc hosting) are not
generally generated from docstrings (though they could be). Folks in
the Python community often use [Sphinx] to generate html docs from
[ReST] -formatted text files.
[Sphinx]: http://sphinx.pocoo.org/
[ReST]: http://docutils.sourceforge.net/rst.html
I think Ruby has something similar for docs at rubyforge.
Incidentally, Perl's CPAN Search http://search.cpan.org/ is of course
the one that extracts and generates docs from embedded (POD)
documentation. It also allows developers to have straight .pod files
in their package which get rendered as html as well.
---John
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Hi all, * Peter Bex peter@xs4all.nl [110303 09:32]: I've written up a proposal for switching our egg repository to a distributed type of deal: http://wiki.call-cc.org/distributed-egg-repos Since many users have indicated they are very much fed up with svn and would prefer to use $FAVORITE_VCS (and some of our existing developers are already doing so, mostly ignoring svn) I think this deserves some serious consideration. I have carried around various thoughts on this matter for a long time now. I hope they are ready for public consumption. Let's restate the pro's of Peters approach: * Proper usage of version control * Distribution of the permission bottleneck * Lower barrier of entry * Reducing checkout and repository size and the cons * Broken links dependencies * Documentation * Additional complexity I would like to adress some fears people already expressed and propose a a bit more than Peter did in his original proposal: Previously some folks mentioned that a DVCS means looser coupling between the 'official' egg repo code and the DVCS contributions with all the risks of being unmaintained, unreachable, converted to python etc... In my little world this can be caused by: * maintainers leaving * stalled development (either WORKS FOR ME or obsolete-d-by-next-hype) * collapse of infrastructure, (sourceforge changed policies, github sold out, etc) * death of domain (spammers taking over, unpaid bills,...) I think the only way to get a grip on these issues is to have a centrally maintained repository or trust other parties to ensure the availability of code. Documentation: At least at the moment I cannot see a drawback in that in either solution. Either the author cares and provides documentation or he doesn't. We do have poorly documented eggs in the repo at the moment and we also received nice documentation from contributors with external main repositories. Additional Complexity: This is also a no brainer, whichever way we decide to go, we will have to change things in the long run. Either to adapt to the larger SVN tree or to the new scheme. Both may be done without breaking existing stuff (apart from path changes of course). So I think the only remaining hard drawback is the looming doom of unmaintained code that I as another egg author would build on my own egg or depend on it as a application developer in my job. So why bother? Because the good parts are really good. Lower entry bars is great! Not restricting people to a certain workflow is also great. I don't mind the checkout size of the repo that much, maybe we can find ways to deliver a functional checkout with some technical helper? But the first two points are the key. I am just not sure if the solution is the right one because of the BROKEN LINKS[tm] scenario. What about the following sketch of an alternative: - Abstract the installation process through the henrietta script as proposed. (I really like this idea). - Automatically *mirror* the contributed archives on call-cc.org, maybe after the code has passed N salmonella rounds of master and experimental branches. - Keep the documentation in the wiki as it is. I envision this mirroring stuff simply as a means of the contributor adding a repo url or something similar to the repo with a branch of it being defined as 'release' or 'stable' code. Then the magical script will pull it in and schedule it for salmonella, mailing back all rrors it finds. After some rounds it is accepted and actively mirrored and registered with henrietta. Of course this also comes with a price tag in terms of our time and effort to get it up and running smoothly. I do think this allows us to combine the best of both worlds though. From perliminary research I beleive that quicklisp works according to this principle. And if there is something new in the lisp library world, it is a central archive and distribution channel. We already do have this advantage, so why not open this up a little for decentral contributions? Thanks for reading this far. I hope I was able to give you a coherent dump of my thoughts on this. May it fertilize the further discussion. Kind regards, Christian ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Christian Kellermann scripsit: - Automatically *mirror* the contributed archives on call-cc.org, maybe after the code has passed N salmonella rounds of master and experimental branches. What is an archive in this context? Is this a .tar.gz of a whole remote repository in some random format (git, hg, bzr, remote svn)? This is the issue that bothers me. Using one DVCS would be great. Using *every* DVCS at the whim of the egg maintainer is an invitation to chaos. And the magic script will in the nature of things only support a subset of all the CVSes out there. -- All Gaul is divided into three parts: the part John Cowan that cooks with lard and goose fat, the parthttp://ccil.org/~cowan that cooks with olive oil, and the part thatco...@ccil.org cooks with butter. --David Chessler ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Hi,
Not sure if I should be posting to this list, however, I thought I'd
take a chance and chime in on 2 points:
1. *Lower Barriers to Entry*. My guess is that the absolute easiest
way for potential contributors to contribute a package is to simply
ask them to upload a tar.gz archive of their project. Supply them with
a handful of requirements, such as:
* name format for the tar.gz file {like so},
* inside the unpacked project dir, its directory structure
should look {like this},
* it should contain a package meta-data file named
{such-and-such}, and the file should contain {the following sorts of
data}.
Then the contributor would not even have to know what sort of
archiving or version control is done on the Chicken/Egg side of
things. And they could use whatever version control they like.
2. *Documentation*. Python's
[Cheeseshop](http://cheeseshop.python.org) has a neat feature where,
if your project has html documentation, you can (manually) upload it
directly to http://packages.python.org/project-name . This way,
contributors aren't forced to use any particular source format for
their docs -- as long as they can generate html from it before
uploading [^1].
In fact, you could go one further and require that packages
contain an `html-doc` dir with at least an index.html file in it. Then
you could trivially automate and eliminate the doc upload step.
[^1]: Incidentally, the [Pandoc](http://johnmacfarlane.net/pandoc/)
tool works well for converting various input doc formats to various
output doc formats (including html).
Just my opinion, but if it's as easy as pie to create and upload a
simple tar.gz package with some html docs (even if it's just a
Markdown-formatted readme converted to html), and if there's a culture
that encourages people to do so, then I think they will.
---John
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
From: John Cowan co...@mercury.ccil.org Subject: Re: [Chicken-hackers] Distributed egg repo proposal Date: Mon, 14 Mar 2011 19:45:20 -0400 Christian Kellermann scripsit: - Automatically *mirror* the contributed archives on call-cc.org, maybe after the code has passed N salmonella rounds of master and experimental branches. What is an archive in this context? Is this a .tar.gz of a whole remote repository in some random format (git, hg, bzr, remote svn)? This is the issue that bothers me. Using one DVCS would be great. Using *every* DVCS at the whim of the egg maintainer is an invitation to chaos. And the magic script will in the nature of things only support a subset of all the CVSes out there. Actually, *every* version of *every* DVCS. Keeping a central store of all contributed repos is essential to have a consistent set of eggs. But it should perhaps only store the raw egg, not the repo (a tarball, say). cheers, felix ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
[^1]: Incidentally, the [Pandoc](http://johnmacfarlane.net/pandoc/) tool works well for converting various input doc formats to various output doc formats (including html). We should make sure not to depend on any external tools. Everything breaks, sooner or later, and we must be able to fix it. cheers, felix ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
On Tue, Mar 15, 2011 at 1:17 AM, Felix fe...@call-with-current-continuation.org wrote: [^1]: Incidentally, the [Pandoc](http://johnmacfarlane.net/pandoc/) tool works well for converting various input doc formats to various output doc formats (including html). We should make sure not to depend on any external tools. Everything breaks, sooner or later, and we must be able to fix it. My point was that, if you tell prospective egg contributors that they can optionally supply their own html docs, they can use whatever tools they want to generate that html (even write it by-hand, if they like). Perhaps I shouldn't have mentioned Pandoc as a specific example tool. There seem to be some other options listed at http://wiki.call-cc.org/chicken-projects/egg-index-4.html#doc-tools . The implication being, if they don't want to document their package in the wiki (perhaps because they've already written their docs in some other format), you could offer them the option of just providing their own raw html and you could install it and host it for them at something like http://eggdocs.call-cc.org/package-name . ---John ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
I've written up a proposal for switching our egg repository to a distributed type of deal: Hello! Thanks again for the thorough proposal. I see that you have put a lot of thought into it and I also believe to understand your motivations - it should be as easy as possible for someone to create and maintain an egg, ideally with the VCS of choice. I personally am quite happy with svn, even though merging is painful sometimes (But I am not a sophisticated VCS user, I quickly get annoyed with tools that try to force any kind of workflow on me. Commit history I find mostly uninteresting, I just don't want to lose work). The current system works and is arguably quite reliable. There are no broken links, and we can do emergency repairs (like trivial syntax errors in some deep dependency's .meta file) anytime. Being centralized, there is always one true HEAD, and having tools like salmonella running over the whole repo is easy. Users will generally not be interested in commit-history or where the stuff is located, they just want to access eggs - anytime, with up-do-date dependencies. Everything maintained and always available. With Alaric's mirror, we even can cope with server outages. Doing disruptive updates (say, a new release branch) can be done (sort of) atomically. Being distributed just for its own sake doesn't buy us anything. WE DON'T WANT BROKEN LINKS. As you correctly point out, a non-existent deep dependency will be horrible indeed (catastrophic, actually). How to avoid broken links or missing eggs without a centralized repository I can't say. Having the egg tree in a git or hg repo would be nice (well, for those that want it, of course). How to map the current layout to (say) git, I don't know. Making it scale is another thing. Currently, eggs can be checked out separately, which should be fine for most egg maintainers. Having a working copy of a whole egg tree or even the whole repo is not strictly necessary. There are tools and VCS-extensions that provide some support for moving history from one VCS to another, so it should be possible to develop in one VCS and then push changes to the central egg repo is possible as well (I don't deny that this may be awkward). If people are uninterested to move eggs or egg releases into our repo, that is of course bad, and making things easy for real or potential contributors is important. What about having a mechanism for automatically importing/updating foreign repos into our tree? Using the mechanisms you describe, not for individual egg accesses (by users, funnelled over henrietta), but for grabbing releases, updating the repo and tagging? That way we could have the best of both worlds: an egg maintainer doesn't have to bother with our repo and can just happily develop, while THE SYSTEM checks for new releases (tags or better: changed release-info files) and pulls the changes. The complexity would be under our control and centralized and the hassles of handling all sorts of VCSes could be dealt with in a central place. So, my opinion is: the current situation is allright for us, good for the user, but mildly inconvenient for some developers. Making things more convenient is something I can understand. But, I must insist on one thing: no broken links. never. cheers, felix ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Distributed egg repo proposal
Felix scripsit: So, my opinion is: the current situation is allright for us, good for the user, but mildly inconvenient for some developers. Making things more convenient is something I can understand. But, I must insist on one thing: no broken links. never. +1, or rather +100. -- John Cowanhttp://ccil.org/~cowanco...@ccil.org [T]here is a Darwinian explanation for the refusal to accept Darwin. Given the very pessimistic conclusions about moral purpose to which his theory drives us, and given the importance of a sense of moral purpose in helping us cope with life, a refusal to believe Darwin's theory may have important survival value. --Ian Johnston ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-hackers
