Re: [racket-dev] updated proposal for moving to packages
On Wed, Jun 5, 2013 at 8:19 PM, Matthew Flatt mfl...@cs.utah.edu wrote: * The details of the repository organization (including where to split repositories) should be different. As described in next section of this message, the experimental repository represents a revised proposal, but there's plenty of room for further refinement. Matthew asked me to try a less fine-grained organization of packages. My initial attempt at this is here: https://github.com/samth/racket/tree/pkg However, I don't think this is an improvement (also, it doesn't fully work). In particular, the dependency cliques are very large, and basically obviate the usefulness of a lot of the splits. There are some small packages, but basically everything depends on everything else. For example, Typed Racket brings in all of the gui libraries, all of the documentation, and the future visualizer. For me, this has been a useful exercise, and I now understand the constraints somewhat better. I think we have, roughly, two options: 1. Something like the split Matthew's tree proposes. In fact, I think we need to split some things further, so that `gui-lib` doesn't depend on scribble-related things. 2. Something much, much more coarse-grained, such as the current split between the 'textual', 'graphical', 'drracket', and 'full' distributions. Note that even these don't really make sense because of documentation build dependencies. I think that 1 is the right choice. I also think that continuing to develop in separate branches as proposals is a mistake. It's very hard to understand what's going on in the `pkg` version of the tree without using it -- I certainly didn't. it's also very hard to construct working trees in this fashion without anyone using the code. If we're going to make this transition soon, we should do it now, and then reorganize packages as necessary. Sam _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
On Thu, Jun 13, 2013 at 3:56 PM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote: As part of my experiment in creating a different split of the repository into packages, I spent some time working with the new setup for building Racket, and cut myself on many of it's rough edges. Some of these are about the package system in general, and some are about the new repository and build infrastructure. * When there's an installation error, it's really hard to tell what state you're in, or what you should do to fix it. Usually, the package manager just says that it's already installed. This is a general issue in the package system, and it's easily the most significant problem I encountered. The package system decides that something is installed very early, and many errors are possible after that. If the package didn't build, or it's dependencies didn't build, or any of many other things happened, the package system thinks everything is ok. The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job. Knowing whether 'raco setup' will succeed after those are in place isn't really possible beforehand and I don't think the idea of 'run raco setup and uninstall if it fails' is robust against the sorts of realistic and innocuous problems that raco setup can report. Maybe you could be more specific about what error you saw and what you wish had happened. * The error message when you look for a missing collection is really long if you have a lot of packages installed * Using parallel make, ie `make -j 8` doesn't work from the new top-level Makefile. * Even though unix systems don't use them, the extra libraries are downloaded and packaged up on unix platforms while building. * When installing a package with a long chain of dependencies, this message gets printed a lot: The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: at-exp-lib syntax-color The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: parser-tools and so on ... We should just collect all the packages to be installed and print them all at once (and perhaps prompt, the way `apt-get` does). Since package dependencies are in the package and not otherwise available, the manager is asking for permission to do a download. After doing the download, it can then look at the dependencies and see if there more. 'apt-get', etc have more monolithic repository structures than our package system, so they can analyze everything locally and compute the transitive closure. Furthermore, 'raco pkg' is prompting you, but you have it set on auto (the default) so it's equivalent to 'yes | apt-get' * I think it's possible to confuse the local catalog by moving directories around in the `pkg` directory. * `make local-catalog` is not incremental -- it always re-downloads and re-packs the external libraries. * Sometimes, errors can be hard to trace to a particular package. For example, this one was because there was a mistake in the future-visualizer package info.rkt file. But I had to go groveling in /var/tmp/ to figure that out. /var/tmp/pkg13711534991371153499937/info.rkt::212: read: expected a closing '' context...: /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:176:17: body /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:173:2: wrap-internal lang:read /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:50:8: temp32 /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:37:0: get-info/full/ext /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:33:0 /home/samth/tmp/pkg-racket/racket/lib/collects/pkg/lib.rkt:1274:0: pkg-single-collection85 Are there other examples? This stack trace is useful to just put a with-handlers at the right spot. Other errors are also hard to track down. In general, it would be very helpful to catch setup errors and report them based on the package being installed and/or the collection being set up. Sam _ Racket Developers list: http://lists.racket-lang.org/dev -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
WRT to the stacktrace below, I guess that if the info.rkt file had been in a suggestively named directory, e.g., /var/tmp/pkg13711534991371153499937/future-visualizer/info.rkt (assuming that the package's name was future-visualizer) that might have been a useful clue. Robby On Thu, Jun 13, 2013 at 9:56 PM, Jay McCarthy jay.mccar...@gmail.comwrote: On Thu, Jun 13, 2013 at 3:56 PM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote: As part of my experiment in creating a different split of the repository into packages, I spent some time working with the new setup for building Racket, and cut myself on many of it's rough edges. Some of these are about the package system in general, and some are about the new repository and build infrastructure. * When there's an installation error, it's really hard to tell what state you're in, or what you should do to fix it. Usually, the package manager just says that it's already installed. This is a general issue in the package system, and it's easily the most significant problem I encountered. The package system decides that something is installed very early, and many errors are possible after that. If the package didn't build, or it's dependencies didn't build, or any of many other things happened, the package system thinks everything is ok. The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job. Knowing whether 'raco setup' will succeed after those are in place isn't really possible beforehand and I don't think the idea of 'run raco setup and uninstall if it fails' is robust against the sorts of realistic and innocuous problems that raco setup can report. Maybe you could be more specific about what error you saw and what you wish had happened. * The error message when you look for a missing collection is really long if you have a lot of packages installed * Using parallel make, ie `make -j 8` doesn't work from the new top-level Makefile. * Even though unix systems don't use them, the extra libraries are downloaded and packaged up on unix platforms while building. * When installing a package with a long chain of dependencies, this message gets printed a lot: The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: at-exp-lib syntax-color The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: parser-tools and so on ... We should just collect all the packages to be installed and print them all at once (and perhaps prompt, the way `apt-get` does). Since package dependencies are in the package and not otherwise available, the manager is asking for permission to do a download. After doing the download, it can then look at the dependencies and see if there more. 'apt-get', etc have more monolithic repository structures than our package system, so they can analyze everything locally and compute the transitive closure. Furthermore, 'raco pkg' is prompting you, but you have it set on auto (the default) so it's equivalent to 'yes | apt-get' * I think it's possible to confuse the local catalog by moving directories around in the `pkg` directory. * `make local-catalog` is not incremental -- it always re-downloads and re-packs the external libraries. * Sometimes, errors can be hard to trace to a particular package. For example, this one was because there was a mistake in the future-visualizer package info.rkt file. But I had to go groveling in /var/tmp/ to figure that out. /var/tmp/pkg13711534991371153499937/info.rkt::212: read: expected a closing '' context...: /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:176:17: body /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:173:2: wrap-internal lang:read /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:50:8: temp32 /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:37:0: get-info/full/ext /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:33:0 /home/samth/tmp/pkg-racket/racket/lib/collects/pkg/lib.rkt:1274:0: pkg-single-collection85 Are there other examples? This stack trace is useful to just put a with-handlers at the right spot. Other errors are also hard to track down. In general, it would be very helpful to catch setup errors and report them based on the package being installed and/or the collection being set up. Sam _ Racket Developers list: http://lists.racket-lang.org/dev -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list:
Re: [racket-dev] experience using the `pkg` branch
Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Robby _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
It does run 'raco setup', it just doesn't have much to do in response to a failure, at least right now. Carl Eastlund On Thu, Jun 13, 2013 at 11:14 PM, Robby Findler ro...@eecs.northwestern.edu wrote: Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Robby _ Racket Developers list: http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
I know it runs it. I don't know why Jay writes The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job.. I can't tell if there's some technical piece I'm missing or not (on the surface, these words sound almost lazy but I *KNOW* Jay is anything but lazy!) Robby On Thu, Jun 13, 2013 at 10:21 PM, Carl Eastlund c...@ccs.neu.edu wrote: It does run 'raco setup', it just doesn't have much to do in response to a failure, at least right now. Carl Eastlund On Thu, Jun 13, 2013 at 11:14 PM, Robby Findler ro...@eecs.northwestern.edu wrote: Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Robby _ Racket Developers list: http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
