[racket-dev] Splitting the Racket repository
As Matthias mentioned in his email a few days ago, we're in the process of splitting the repository so that it doesn't bundle together so many packages. I've started this process already, and a number of packages have already been split out. For most people, this won't have a big impact, but I'll outline changes for different sorts of people below, and mention a few open questions. All the history for the code has been preserved, and for code that dates back before 2005, the history is extended back to the original CVS repository. See https://github.com/racket/games/ for an example of this. # Changes for other users If you use Racket by downloading a release or a snapshot, nothing will change for you. # Changes for git users If you build Racket from source from Git, that build now contains fewer packages. There is not yet an single-step way to get all of the split pkgs as git repositories; we plan to write a script for this soon. To clone individual repositories, use the new `--clone` option for `raco pkg`, such as: raco pkg install --clone pkg-build git://github.com/racket/pkg-build or for packages that are grouped together in a single repository raco pkg install --clone remote-shell git://github.com/racket/remote-shell?path=remote-shell-lib git://github.com/racket/remote-shell?path=remote-shell git://github.com/racket/remote-shell?path=remote-shell-doc Note that the clones created by `raco pkg install` cannot be pushed to with the default origin remote. # Changes for committers If you have commit access to the main Racket repository, you'll be added to the `racket` organization on GitHub, which will give you access to commit to the repositories there, which is where all of the new packages are. # Open issues ## DrDr Currently, DrDr builds all the split packages as well as the current repository. But it doesn't yet notice when changes are pushed to the split packages. Also, it doesn't yet know who is responsible for code in split packages, so if some of your code in such as package breaks, you won't get an email. We plan to fix both of these issues soon. Also, DrDr now builds several packages that it didn't before: the s3-sync, sha, http, and aws packages. Currently, these packages have some test failures, but we hope to fix that soon. ## Determining the content of the distribution Previously, the content of the distribution was the dependencies of the main-distribution package. Since that's in the repository, it can't depend on the split packages. Thus, we need a new canonical home for this information. ## Release process This will also require changes to the release process, which I'll leave to the release managers to describe. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Splitting the Racket repository
On Sat, Nov 29, 2014 at 7:14 PM, Sam Tobin-Hochstadt sa...@cs.indiana.edu wrote: All the history for the code has been preserved, and for code that dates back before 2005, the history is extended back to the original CVS repository. See https://github.com/racket/games/ for an example of this. There's a failure in the import -- if you look at the commits of this repo (https://github.com/racket/games/commits/master), then starting from Robby's commit from Jul 17 (add a contract to make-card...) and going back, the original commit references are all bogus. If you build Racket from source from Git, that build now contains fewer packages. There is not yet an single-step way to get all of the split pkgs as git repositories; we plan to write a script for this soon. Any reason they are not git modules? (They've improved a lot, and they're even supported in github as links as long as they point to other GH repos. See for example: https://github.com/elibarzilay/test) To clone individual repositories, use the new `--clone` option for `raco pkg`, such as: raco pkg install --clone pkg-build git://github.com/racket/pkg-build or for packages that are grouped together in a single repository raco pkg install --clone remote-shell git://github.com/racket/remote-shell?path=remote-shell-lib git://github.com/racket/remote-shell?path=remote-shell git://github.com/racket/remote-shell?path=remote-shell-doc Note that the clones created by `raco pkg install` cannot be pushed to with the default origin remote. This is very obscure. Is there a compact description of what to do when you want to make a change in a file that is not on the main repo? (Hopefully there's obvious agreement on the need for this to be compact.) -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Splitting the Racket repository
On Sat, Nov 29, 2014 at 8:16 PM, Eli Barzilay e...@barzilay.org wrote: On Sat, Nov 29, 2014 at 7:14 PM, Sam Tobin-Hochstadt sa...@cs.indiana.edu wrote: All the history for the code has been preserved, and for code that dates back before 2005, the history is extended back to the original CVS repository. See https://github.com/racket/games/ for an example of this. There's a failure in the import -- if you look at the commits of this repo (https://github.com/racket/games/commits/master), then starting from Robby's commit from Jul 17 (add a contract to make-card...) and going back, the original commit references are all bogus. For packages with old history, the shas are in reference to https://github.com/samth/old-plt-new/ (which should maybe have a better name). If you build Racket from source from Git, that build now contains fewer packages. There is not yet an single-step way to get all of the split pkgs as git repositories; we plan to write a script for this soon. Any reason they are not git modules? (They've improved a lot, and they're even supported in github as links as long as they point to other GH repos. See for example: https://github.com/elibarzilay/test) The goal is to have packages that are in the main distribution not have a particular special status, beyond being in the list of things that gets put in the distribution. Also, a situation where you have to update two things when you do a commit is not ideal. To clone individual repositories, use the new `--clone` option for `raco pkg`, such as: raco pkg install --clone pkg-build git://github.com/racket/pkg-build or for packages that are grouped together in a single repository raco pkg install --clone remote-shell git://github.com/racket/remote-shell?path=remote-shell-lib git://github.com/racket/remote-shell?path=remote-shell git://github.com/racket/remote-shell?path=remote-shell-doc Note that the clones created by `raco pkg install` cannot be pushed to with the default origin remote. This is very obscure. Is there a compact description of what to do when you want to make a change in a file that is not on the main repo? You just need to edit and change the relevant repository -- nothing else is required. You don't have to use --clone or any other mechanism. Note that the command line above can be shortened to: raco pkg install --clone remote-shell remote-shell-lib remote-shell-doc remote-shell (that `remote-shell` appears twice there is intentional). Sam _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Splitting the Racket repository
On Sat, Nov 29, 2014 at 8:30 PM, Sam Tobin-Hochstadt sa...@cs.indiana.edu wrote: On Sat, Nov 29, 2014 at 8:16 PM, Eli Barzilay e...@barzilay.org wrote: On Sat, Nov 29, 2014 at 7:14 PM, Sam Tobin-Hochstadt sa...@cs.indiana.edu wrote: All the history for the code has been preserved, and for code that dates back before 2005, the history is extended back to the original CVS repository. See https://github.com/racket/games/ for an example of this. There's a failure in the import -- if you look at the commits of this repo (https://github.com/racket/games/commits/master), then starting from Robby's commit from Jul 17 (add a contract to make-card...) and going back, the original commit references are all bogus. For packages with old history, the shas are in reference to https://github.com/samth/old-plt-new/ (which should maybe have a better name). Yeah, very bad name but an even worse location... Since this is a major reorganization it would also be a reasonable to do a forced push of that thing to the main repository so the references make sense. (This is assuming that you cannot refer to the proper commits in the current repo for some reason.) If all else fails, you should at least include in the commits some pointer to a repo that has it. (Sounds picky, but having no reliable history is one thing that can make a company avoid using it.) Any reason they are not git modules? (They've improved a lot, and they're even supported in github as links as long as they point to other GH repos. See for example: https://github.com/elibarzilay/test) The goal is to have packages that are in the main distribution not have a particular special status, beyond being in the list of things that gets put in the distribution. I'm not talking about the distribution here -- just some meta repository that has everything in it in a convenient way for the people who have code in there. (And assuming that this meta repository will build with all of these packages that still not be related to a distribution.) (A different issue is organizing distributions: at some point in the past I pointed at the fact that git modules could be used for this, essentially being a collection of pointers to code that is included. But that bus has long gone, so I'm definitely not talking about that.) Also, a situation where you have to update two things when you do a commit is not ideal. I absolutely agree with that. Probably even more than you do, and certainly for a much longer time (that's the essense of my bundle-related rants years ago about spaghetti dependencies). I suppose that you're worrying about modules making it easy to do such cross-repo commits -- but it's no easier than just having a bunch of repos and doing a commit over them simultaneously. So it's as doable, and in both cases will be awkward enough to make it clear to the committer that something is wrong. To clone individual repositories, use the new `--clone` option for `raco pkg`, such as: [...] This is very obscure. Is there a compact description of what to do when you want to make a change in a file that is not on the main repo? You just need to edit and change the relevant repository -- nothing else is required. You don't have to use --clone or any other mechanism. [...] This doesn't answer my question -- so I think that I wasn't clear enough. Say that I have some change to a specific file. I want to know how to do it in the following two ways: 1. Get the equivalent of the whole distribution in the form of a bunch of repositories, including the repository that I want to change. I make the change in the file, re-build, test things out, and eventually push. This is the way that is more relevant for internal people, and I suppose that there would be some script that will do all of that. (The equivalent of a toplevel `make'). 2. The thing that is more relevant for others (and that's the important one): I have a normal installation, I find a bug -- what do I do now? I can't just report the bug, the file, and my patch since that's using a path in the distribution that is mostly unrelated to the source repo for the file. Ideally, there would be some way to turn some containing directory into a repository so I can commit my change, and push it to a fork to do a PR. But that's a little extreme -- so what I'm talking about is more like a way to say: get this repository and use it instead of the stuff in my distribution, so I can work and push from that. I'm guessing that this is the intention behind that command, but it's obscure since it's not obvious what to do. (And obviousness is important to encourage contributions.) As a concrete example: I find a bug in a file like share/pkgs/games/gcalc/gcalc.rkt and want to fork, edit, and push a fix. What do I do? -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay:
