Re: [racket-dev] experience using the `pkg` branch
At Thu, 13 Jun 2013 17:56:52 -0400, Sam Tobin-Hochstadt wrote: * The error message when you look for a missing collection is really long if you have a lot of packages installed Yes. I have been thinking about whether there's a better solution than just not showing the paths, but I can just drop the path list for now. * Using parallel make, ie `make -j 8` doesn't work from the new top-level Makefile. Will fix. I think this is just a matter of use $(MAKE) from the top-level Makefile. * Even though unix systems don't use them, the extra libraries are downloaded and packaged up on unix platforms while building. I think you're referring to the `server' Makefile target, which is intended for a build process where the libraries will be needed by clients. In any case, the process for dealing with native libraries will likely change/improve when we have those packages available from a catalog server. * I think it's possible to confuse the local catalog by moving directories around in the `pkg` directory. Definitely. You have to rebuild the local catalog after moving things around. * `make local-catalog` is not incremental -- it always re-downloads and re-packs the external libraries. True. As with the preceding two bullets, this can improve as we figure out how we want to work. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
On Thu, Jun 13, 2013 at 10:56 PM, Jay McCarthy jay.mccar...@gmail.com wrote: On Thu, Jun 13, 2013 at 3:56 PM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote: As part of my experiment in creating a different split of the repository into packages, I spent some time working with the new setup for building Racket, and cut myself on many of it's rough edges. Some of these are about the package system in general, and some are about the new repository and build infrastructure. * When there's an installation error, it's really hard to tell what state you're in, or what you should do to fix it. Usually, the package manager just says that it's already installed. This is a general issue in the package system, and it's easily the most significant problem I encountered. The package system decides that something is installed very early, and many errors are possible after that. If the package didn't build, or it's dependencies didn't build, or any of many other things happened, the package system thinks everything is ok. The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job. Knowing whether 'raco setup' will succeed after those are in place isn't really possible beforehand and I don't think the idea of 'run raco setup and uninstall if it fails' is robust against the sorts of realistic and innocuous problems that raco setup can report. Maybe you could be more specific about what error you saw and what you wish had happened. In addition to the larger point Robby makes, this can be pretty confusing. For example, you can fail to install enough dependencies, I think. Another problem is that there's no way to know what to do to fix things. Say there's an error in `raco setup` that's transient (the machine lost power, for example). What command do I run to 'fix' the setup? I don't even know what collections were being installed. * The error message when you look for a missing collection is really long if you have a lot of packages installed * Using parallel make, ie `make -j 8` doesn't work from the new top-level Makefile. * Even though unix systems don't use them, the extra libraries are downloaded and packaged up on unix platforms while building. * When installing a package with a long chain of dependencies, this message gets printed a lot: The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: at-exp-lib syntax-color The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: parser-tools and so on ... We should just collect all the packages to be installed and print them all at once (and perhaps prompt, the way `apt-get` does). Since package dependencies are in the package and not otherwise available, the manager is asking for permission to do a download. After doing the download, it can then look at the dependencies and see if there more. 'apt-get', etc have more monolithic repository structures than our package system, so they can analyze everything locally and compute the transitive closure. Furthermore, 'raco pkg' is prompting you, but you have it set on auto (the default) so it's equivalent to 'yes | apt-get' Ah, that makes a lot of sense. Could we do something different in auto mode to just print out the packages that were installed at the end? * I think it's possible to confuse the local catalog by moving directories around in the `pkg` directory. * `make local-catalog` is not incremental -- it always re-downloads and re-packs the external libraries. * Sometimes, errors can be hard to trace to a particular package. For example, this one was because there was a mistake in the future-visualizer package info.rkt file. But I had to go groveling in /var/tmp/ to figure that out. /var/tmp/pkg13711534991371153499937/info.rkt::212: read: expected a closing '' context...: /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:176:17: body /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:173:2: wrap-internal lang:read /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:50:8: temp32 /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:37:0: get-info/full/ext /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:33:0 /home/samth/tmp/pkg-racket/racket/lib/collects/pkg/lib.rkt:1274:0: pkg-single-collection85 Are there other examples? This stack trace is useful to just put a with-handlers at the right spot. I got plenty of `raco setup` errors that were hard to track down in the same way. Those don't have a stacktrace, though. I can't remember any others that were like this one. Sam _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
True, but in this case, the code that is doing that reading knows what the package was and could give a good error message. Jay On Thu, Jun 13, 2013 at 9:13 PM, Robby Findler ro...@eecs.northwestern.edu wrote: WRT to the stacktrace below, I guess that if the info.rkt file had been in a suggestively named directory, e.g., /var/tmp/pkg13711534991371153499937/future-visualizer/info.rkt (assuming that the package's name was future-visualizer) that might have been a useful clue. Robby On Thu, Jun 13, 2013 at 9:56 PM, Jay McCarthy jay.mccar...@gmail.com wrote: On Thu, Jun 13, 2013 at 3:56 PM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote: As part of my experiment in creating a different split of the repository into packages, I spent some time working with the new setup for building Racket, and cut myself on many of it's rough edges. Some of these are about the package system in general, and some are about the new repository and build infrastructure. * When there's an installation error, it's really hard to tell what state you're in, or what you should do to fix it. Usually, the package manager just says that it's already installed. This is a general issue in the package system, and it's easily the most significant problem I encountered. The package system decides that something is installed very early, and many errors are possible after that. If the package didn't build, or it's dependencies didn't build, or any of many other things happened, the package system thinks everything is ok. The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job. Knowing whether 'raco setup' will succeed after those are in place isn't really possible beforehand and I don't think the idea of 'run raco setup and uninstall if it fails' is robust against the sorts of realistic and innocuous problems that raco setup can report. Maybe you could be more specific about what error you saw and what you wish had happened. * The error message when you look for a missing collection is really long if you have a lot of packages installed * Using parallel make, ie `make -j 8` doesn't work from the new top-level Makefile. * Even though unix systems don't use them, the extra libraries are downloaded and packaged up on unix platforms while building. * When installing a package with a long chain of dependencies, this message gets printed a lot: The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: at-exp-lib syntax-color The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: parser-tools and so on ... We should just collect all the packages to be installed and print them all at once (and perhaps prompt, the way `apt-get` does). Since package dependencies are in the package and not otherwise available, the manager is asking for permission to do a download. After doing the download, it can then look at the dependencies and see if there more. 'apt-get', etc have more monolithic repository structures than our package system, so they can analyze everything locally and compute the transitive closure. Furthermore, 'raco pkg' is prompting you, but you have it set on auto (the default) so it's equivalent to 'yes | apt-get' * I think it's possible to confuse the local catalog by moving directories around in the `pkg` directory. * `make local-catalog` is not incremental -- it always re-downloads and re-packs the external libraries. * Sometimes, errors can be hard to trace to a particular package. For example, this one was because there was a mistake in the future-visualizer package info.rkt file. But I had to go groveling in /var/tmp/ to figure that out. /var/tmp/pkg13711534991371153499937/info.rkt::212: read: expected a closing '' context...: /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:176:17: body /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:173:2: wrap-internal lang:read /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:50:8: temp32 /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:37:0: get-info/full/ext /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:33:0 /home/samth/tmp/pkg-racket/racket/lib/collects/pkg/lib.rkt:1274:0: pkg-single-collection85 Are there other examples? This stack trace is useful to just put a with-handlers at the right spot. Other errors are also hard to track down. In general, it would be very helpful to catch setup errors and report them based on the package being installed and/or the collection being set up. Sam _ Racket Developers list: http://lists.racket-lang.org/dev --
Re: [racket-dev] experience using the `pkg` branch
On Fri, Jun 14, 2013 at 6:19 AM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote: On Thu, Jun 13, 2013 at 10:56 PM, Jay McCarthy jay.mccar...@gmail.com wrote: On Thu, Jun 13, 2013 at 3:56 PM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote: * The error message when you look for a missing collection is really long if you have a lot of packages installed * Using parallel make, ie `make -j 8` doesn't work from the new top-level Makefile. * Even though unix systems don't use them, the extra libraries are downloaded and packaged up on unix platforms while building. * When installing a package with a long chain of dependencies, this message gets printed a lot: The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: at-exp-lib syntax-color The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: parser-tools and so on ... We should just collect all the packages to be installed and print them all at once (and perhaps prompt, the way `apt-get` does). Since package dependencies are in the package and not otherwise available, the manager is asking for permission to do a download. After doing the download, it can then look at the dependencies and see if there more. 'apt-get', etc have more monolithic repository structures than our package system, so they can analyze everything locally and compute the transitive closure. Furthermore, 'raco pkg' is prompting you, but you have it set on auto (the default) so it's equivalent to 'yes | apt-get' Ah, that makes a lot of sense. Could we do something different in auto mode to just print out the packages that were installed at the end? Yes, that's a good idea. I'll do that. * I think it's possible to confuse the local catalog by moving directories around in the `pkg` directory. * `make local-catalog` is not incremental -- it always re-downloads and re-packs the external libraries. * Sometimes, errors can be hard to trace to a particular package. For example, this one was because there was a mistake in the future-visualizer package info.rkt file. But I had to go groveling in /var/tmp/ to figure that out. /var/tmp/pkg13711534991371153499937/info.rkt::212: read: expected a closing '' context...: /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:176:17: body /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:173:2: wrap-internal lang:read /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:50:8: temp32 /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:37:0: get-info/full/ext /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:33:0 /home/samth/tmp/pkg-racket/racket/lib/collects/pkg/lib.rkt:1274:0: pkg-single-collection85 Are there other examples? This stack trace is useful to just put a with-handlers at the right spot. I got plenty of `raco setup` errors that were hard to track down in the same way. Those don't have a stacktrace, though. I can't remember any others that were like this one. Sam -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
I'll try to respond to these four messages at the same time... Sam said, In addition to the larger point Robby makes, this can be pretty confusing. For example, you can fail to install enough dependencies, I think. Another problem is that there's no way to know what to do to fix things. Say there's an error in `raco setup` that's transient (the machine lost power, for example). What command do I run to 'fix' the setup? I don't even know what collections were being installed. On Thu, Jun 13, 2013 at 9:27 PM, Robby Findler ro...@eecs.northwestern.edu wrote: I know it runs it. I don't know why Jay writes The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job.. I can't tell if there's some technical piece I'm missing or not (on the surface, these words sound almost lazy but I *KNOW* Jay is anything but lazy!) Robby On Thu, Jun 13, 2013 at 10:21 PM, Carl Eastlund c...@ccs.neu.edu wrote: It does run 'raco setup', it just doesn't have much to do in response to a failure, at least right now. Carl Eastlund On Thu, Jun 13, 2013 at 11:14 PM, Robby Findler ro...@eecs.northwestern.edu wrote: Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Short: I interpret these messages to be saying, raco pkg printed error messages and didn't say what was wrong or what I should do about it. The messages came from 'raco setup', of course, and 'raco pkg' gave no interpretation. As a simple first step, we can have 'raco pkg' observe the output of 'raco setup' (which might require changing it) and if it gives any error message of any kind, end with a message like: raco pkg detected strange output during setup. This could mean there was a problem. If you would like to undo this command, run: raco pkg remove the-package-you-installed This sounds like a simple to change to make, but there's one big problem. After 'raco pkg update' runs, the undo command is much more complicated and scary in its own way raco pkg remove --force the-package-you-updated raco pkg install the-source-for-the-package-you-updated-at-the-old-git-commit But some package sources may not be possible to get back, such as an old tar ball that you downloaded and don't have any more. When raco pkg update runs normally, it leaves the old files in place until the end and then deletes them. It could put these into some sort of 'archive' location to facilitate that. But it's a big change and easy to get very wrong. Long: I also interpret these messages to be saying When raco pkg returns with exit status 0, the packages I asked to be install/updated should 'work'. This is a reasonable wish, but I don't think it can be enforced technically. Let's un-ravel worked-ness: 1. Does the task I thought it would do Clearly no way to ensure this. No package system does this. 2. Every file executes without errors This is what DrDr normally does and is too burdensome for every package, and includes problems of the next stage. No package system does this. 3. Passes its test cases on my system CPAN is fairly unique in doing this upon install, but virtually no other package system does this. I don't think it is actually useful, because there's no reason to have confidence in the test cases and it's hard from a test case writer perspective to get everything in place. (For example, a Mongo DB interface needs a running Mongo DB server to test against.) 4. Compiles on my system Most package systems don't need to think about this, because they ship only binaries. Languages like Perl and Javascript also don't really do this, because they don't compile in any real sense. Languages that do can typically make a pretty simple promise based on their languages inability to have bizarre and unpredicatable compile-time dependencies. Racket has the ability to provide bizarre and unpredictable compile-time dependencies: #lang racket/base (define-syntax (macro-invocation stx) (if matthew-isn't-looking #'(require #,(random-file)) #'(void))) (macro-invocation) Since we can't really offer this, the best we can do is trust the dependencies listed. We have ideas on how to enforce those dependencies, but it isn't implemented yet. 5. Its actual dependencies are on my system This is like the above. 6. Its listed dependencies are on my system This is the standard that 'raco pkg' uses. It guarantees that either all dependencies are installed or that no changes were made. This means that you can never run 'raco pkg update', download a new version of an installed package that has a new dependency that conflicts with a current package, and then end up with a failure
Re: [racket-dev] experience using the `pkg` branch
Jay, Thanks for the detailed summary. Right now I think the biggest missing piece of information when something goes wrong is which packages were affected?. If I download package A, and it pulls in B and C, and they pull in D, E, and F, and then somewhere along the lines raco setup fails, I now have six packages installed, and no clear idea which ones are working fine and which ones need to be fixed. If raco pkg could communicate with raco setup in some way to associate files that failed with the packages they came from, it would be very useful at the end to get a report like: raco pkg successfully installed packages B, E, and F; raco setup reports errors with the installation of A, C, and D. That or a raco pkg status command to look up similar information. Otherwise the user is left to track down the dependencies responsible for errors themselves, which can very time consuming when each step of the search involves running raco setup on some new subset of packages. I'm sure this extra scrutiny and criticism for the package system isn't easy, so let me take this moment to also say -- thanks so much for writing the package system! It wouldn't be worth all this nitpicking if it weren't a great system; we've come a long way from distributing source code via .plt files, to having something worth using to bundle up our core software. Carl Eastlund On Fri, Jun 14, 2013 at 10:13 AM, Jay McCarthy jay.mccar...@gmail.comwrote: I'll try to respond to these four messages at the same time... Sam said, In addition to the larger point Robby makes, this can be pretty confusing. For example, you can fail to install enough dependencies, I think. Another problem is that there's no way to know what to do to fix things. Say there's an error in `raco setup` that's transient (the machine lost power, for example). What command do I run to 'fix' the setup? I don't even know what collections were being installed. On Thu, Jun 13, 2013 at 9:27 PM, Robby Findler ro...@eecs.northwestern.edu wrote: I know it runs it. I don't know why Jay writes The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job.. I can't tell if there's some technical piece I'm missing or not (on the surface, these words sound almost lazy but I *KNOW* Jay is anything but lazy!) Robby On Thu, Jun 13, 2013 at 10:21 PM, Carl Eastlund c...@ccs.neu.edu wrote: It does run 'raco setup', it just doesn't have much to do in response to a failure, at least right now. Carl Eastlund On Thu, Jun 13, 2013 at 11:14 PM, Robby Findler ro...@eecs.northwestern.edu wrote: Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Short: I interpret these messages to be saying, raco pkg printed error messages and didn't say what was wrong or what I should do about it. The messages came from 'raco setup', of course, and 'raco pkg' gave no interpretation. As a simple first step, we can have 'raco pkg' observe the output of 'raco setup' (which might require changing it) and if it gives any error message of any kind, end with a message like: raco pkg detected strange output during setup. This could mean there was a problem. If you would like to undo this command, run: raco pkg remove the-package-you-installed This sounds like a simple to change to make, but there's one big problem. After 'raco pkg update' runs, the undo command is much more complicated and scary in its own way raco pkg remove --force the-package-you-updated raco pkg install the-source-for-the-package-you-updated-at-the-old-git-commit But some package sources may not be possible to get back, such as an old tar ball that you downloaded and don't have any more. When raco pkg update runs normally, it leaves the old files in place until the end and then deletes them. It could put these into some sort of 'archive' location to facilitate that. But it's a big change and easy to get very wrong. Long: I also interpret these messages to be saying When raco pkg returns with exit status 0, the packages I asked to be install/updated should 'work'. This is a reasonable wish, but I don't think it can be enforced technically. Let's un-ravel worked-ness: 1. Does the task I thought it would do Clearly no way to ensure this. No package system does this. 2. Every file executes without errors This is what DrDr normally does and is too burdensome for every package, and includes problems of the next stage. No package system does this. 3. Passes its test cases on my system CPAN is fairly unique in doing this upon install, but virtually no other package system does this. I don't think
Re: [racket-dev] experience using the `pkg` branch
On Fri, Jun 14, 2013 at 8:40 AM, Carl Eastlund c...@ccs.neu.edu wrote: Jay, Thanks for the detailed summary. Right now I think the biggest missing piece of information when something goes wrong is which packages were affected?. If I download package A, and it pulls in B and C, and they pull in D, E, and F, and then somewhere along the lines raco setup fails, I now have six packages installed, and no clear idea which ones are working fine and which ones need to be fixed. If raco pkg could communicate with raco setup in some way to associate files that failed with the packages they came from, it would be very useful at the end to get a report like: raco pkg successfully installed packages B, E, and F; raco setup reports errors with the installation of A, C, and D. That or a raco pkg status command to look up similar information. Otherwise the user is left to track down the dependencies responsible for errors themselves, which can very time consuming when each step of the search involves running raco setup on some new subset of packages. One of my vices is thinking of the simplest way to do something and not the best for the user. One really simple way to do this is to make it so that rather than running one 'raco setup' after a whole bunch of package activity, it does many and can say what happened after each one. This is more like Planet. I think it would be slightly slower. Another idea is to make it so that 'raco setup' can have a mode where it passes error structures back that contain the paths (so 'raco pkg' knows where they are and which package directory it was) rather than just printing error messages. I'm not sure how 'raco setup' is internally organized enough to know if that is feasible. I'm sure this extra scrutiny and criticism for the package system isn't easy, so let me take this moment to also say -- thanks so much for writing the package system! It wouldn't be worth all this nitpicking if it weren't a great system; we've come a long way from distributing source code via .plt files, to having something worth using to bundle up our core software. Carl Eastlund On Fri, Jun 14, 2013 at 10:13 AM, Jay McCarthy jay.mccar...@gmail.com wrote: I'll try to respond to these four messages at the same time... Sam said, In addition to the larger point Robby makes, this can be pretty confusing. For example, you can fail to install enough dependencies, I think. Another problem is that there's no way to know what to do to fix things. Say there's an error in `raco setup` that's transient (the machine lost power, for example). What command do I run to 'fix' the setup? I don't even know what collections were being installed. On Thu, Jun 13, 2013 at 9:27 PM, Robby Findler ro...@eecs.northwestern.edu wrote: I know it runs it. I don't know why Jay writes The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job.. I can't tell if there's some technical piece I'm missing or not (on the surface, these words sound almost lazy but I *KNOW* Jay is anything but lazy!) Robby On Thu, Jun 13, 2013 at 10:21 PM, Carl Eastlund c...@ccs.neu.edu wrote: It does run 'raco setup', it just doesn't have much to do in response to a failure, at least right now. Carl Eastlund On Thu, Jun 13, 2013 at 11:14 PM, Robby Findler ro...@eecs.northwestern.edu wrote: Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Short: I interpret these messages to be saying, raco pkg printed error messages and didn't say what was wrong or what I should do about it. The messages came from 'raco setup', of course, and 'raco pkg' gave no interpretation. As a simple first step, we can have 'raco pkg' observe the output of 'raco setup' (which might require changing it) and if it gives any error message of any kind, end with a message like: raco pkg detected strange output during setup. This could mean there was a problem. If you would like to undo this command, run: raco pkg remove the-package-you-installed This sounds like a simple to change to make, but there's one big problem. After 'raco pkg update' runs, the undo command is much more complicated and scary in its own way raco pkg remove --force the-package-you-updated raco pkg install the-source-for-the-package-you-updated-at-the-old-git-commit But some package sources may not be possible to get back, such as an old tar ball that you downloaded and don't have any more. When raco pkg update runs normally, it leaves the old files in place until the end and then deletes them. It could put these into some sort of 'archive' location to facilitate
Re: [racket-dev] experience using the `pkg` branch
On Jun 14, 2013, at 10:49 AM, Jay McCarthy wrote: Another idea is to make it so that 'raco setup' can have a mode where it passes error structures back that contain the paths (so 'raco pkg' knows where they are and which package directory it was) rather than just printing error messages. I'm not sure how 'raco setup' is internally organized enough to know if that is feasible. This sounds preferable, even if it is a lot more work. -- Matthias _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
Thanks, Jay. I have two high-level negative comments. 1) I don't think that it is feasible to aim for a world where 'raco setup' never errors. (The real-world variation of your matthew-isnt-looking example is system-specific code and we do do that a bunch.) 2) You are currently making a distinction between 'raco pkg' and 'raco setup' as if these were the only two things. But this is actually unfair. Really, when you are at this level of granularity, 'raco pkg' may actually be one thing but 'raco setup' is actually N things (zo file building, info.rkt cache building, documentation building, documentation index building (a different thing) executable creation, and some stuff I'm forgetting (oh: and all in parallel and on a per-collection basis, etc etc)). Despite that, we have hammered on 'raco setup' enough that it now, mostly, seems to be a single thing that succeeds or fails in a somewhat understandable way for all kinds of common failures. At least that's one conclusion I take from earlier messages in this thread. :) That is where I'd like to us to aim for raco pkg to be. I think this is about smoothing out edges in common error situations and not about fundamental changes. On the positive side, I really like how you can be precise about your guarantee #6. This seems like a fantastic foundation on which to build a system where 'raco pkg' might trigger a 'raco setup' that spews error messages but that various queries afterwards can help the user explore what has happened and how to fix it. I appreciate your stubbornness on keeping the internal/logical integrity of the pkg system. We need that. Robby On Fri, Jun 14, 2013 at 9:13 AM, Jay McCarthy jay.mccar...@gmail.comwrote: I'll try to respond to these four messages at the same time... Sam said, In addition to the larger point Robby makes, this can be pretty confusing. For example, you can fail to install enough dependencies, I think. Another problem is that there's no way to know what to do to fix things. Say there's an error in `raco setup` that's transient (the machine lost power, for example). What command do I run to 'fix' the setup? I don't even know what collections were being installed. On Thu, Jun 13, 2013 at 9:27 PM, Robby Findler ro...@eecs.northwestern.edu wrote: I know it runs it. I don't know why Jay writes The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job.. I can't tell if there's some technical piece I'm missing or not (on the surface, these words sound almost lazy but I *KNOW* Jay is anything but lazy!) Robby On Thu, Jun 13, 2013 at 10:21 PM, Carl Eastlund c...@ccs.neu.edu wrote: It does run 'raco setup', it just doesn't have much to do in response to a failure, at least right now. Carl Eastlund On Thu, Jun 13, 2013 at 11:14 PM, Robby Findler ro...@eecs.northwestern.edu wrote: Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Short: I interpret these messages to be saying, raco pkg printed error messages and didn't say what was wrong or what I should do about it. The messages came from 'raco setup', of course, and 'raco pkg' gave no interpretation. As a simple first step, we can have 'raco pkg' observe the output of 'raco setup' (which might require changing it) and if it gives any error message of any kind, end with a message like: raco pkg detected strange output during setup. This could mean there was a problem. If you would like to undo this command, run: raco pkg remove the-package-you-installed This sounds like a simple to change to make, but there's one big problem. After 'raco pkg update' runs, the undo command is much more complicated and scary in its own way raco pkg remove --force the-package-you-updated raco pkg install the-source-for-the-package-you-updated-at-the-old-git-commit But some package sources may not be possible to get back, such as an old tar ball that you downloaded and don't have any more. When raco pkg update runs normally, it leaves the old files in place until the end and then deletes them. It could put these into some sort of 'archive' location to facilitate that. But it's a big change and easy to get very wrong. Long: I also interpret these messages to be saying When raco pkg returns with exit status 0, the packages I asked to be install/updated should 'work'. This is a reasonable wish, but I don't think it can be enforced technically. Let's un-ravel worked-ness: 1. Does the task I thought it would do Clearly no way to ensure this. No package system does this. 2. Every file executes without errors This is what DrDr normally does and is too
Re: [racket-dev] experience using the `pkg` branch
On Thu, Jun 13, 2013 at 3:56 PM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote: As part of my experiment in creating a different split of the repository into packages, I spent some time working with the new setup for building Racket, and cut myself on many of it's rough edges. Some of these are about the package system in general, and some are about the new repository and build infrastructure. * When there's an installation error, it's really hard to tell what state you're in, or what you should do to fix it. Usually, the package manager just says that it's already installed. This is a general issue in the package system, and it's easily the most significant problem I encountered. The package system decides that something is installed very early, and many errors are possible after that. If the package didn't build, or it's dependencies didn't build, or any of many other things happened, the package system thinks everything is ok. The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job. Knowing whether 'raco setup' will succeed after those are in place isn't really possible beforehand and I don't think the idea of 'run raco setup and uninstall if it fails' is robust against the sorts of realistic and innocuous problems that raco setup can report. Maybe you could be more specific about what error you saw and what you wish had happened. * The error message when you look for a missing collection is really long if you have a lot of packages installed * Using parallel make, ie `make -j 8` doesn't work from the new top-level Makefile. * Even though unix systems don't use them, the extra libraries are downloaded and packaged up on unix platforms while building. * When installing a package with a long chain of dependencies, this message gets printed a lot: The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: at-exp-lib syntax-color The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: parser-tools and so on ... We should just collect all the packages to be installed and print them all at once (and perhaps prompt, the way `apt-get` does). Since package dependencies are in the package and not otherwise available, the manager is asking for permission to do a download. After doing the download, it can then look at the dependencies and see if there more. 'apt-get', etc have more monolithic repository structures than our package system, so they can analyze everything locally and compute the transitive closure. Furthermore, 'raco pkg' is prompting you, but you have it set on auto (the default) so it's equivalent to 'yes | apt-get' * I think it's possible to confuse the local catalog by moving directories around in the `pkg` directory. * `make local-catalog` is not incremental -- it always re-downloads and re-packs the external libraries. * Sometimes, errors can be hard to trace to a particular package. For example, this one was because there was a mistake in the future-visualizer package info.rkt file. But I had to go groveling in /var/tmp/ to figure that out. /var/tmp/pkg13711534991371153499937/info.rkt::212: read: expected a closing '' context...: /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:176:17: body /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:173:2: wrap-internal lang:read /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:50:8: temp32 /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:37:0: get-info/full/ext /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:33:0 /home/samth/tmp/pkg-racket/racket/lib/collects/pkg/lib.rkt:1274:0: pkg-single-collection85 Are there other examples? This stack trace is useful to just put a with-handlers at the right spot. Other errors are also hard to track down. In general, it would be very helpful to catch setup errors and report them based on the package being installed and/or the collection being set up. Sam _ Racket Developers list: http://lists.racket-lang.org/dev -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
WRT to the stacktrace below, I guess that if the info.rkt file had been in a suggestively named directory, e.g., /var/tmp/pkg13711534991371153499937/future-visualizer/info.rkt (assuming that the package's name was future-visualizer) that might have been a useful clue. Robby On Thu, Jun 13, 2013 at 9:56 PM, Jay McCarthy jay.mccar...@gmail.comwrote: On Thu, Jun 13, 2013 at 3:56 PM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote: As part of my experiment in creating a different split of the repository into packages, I spent some time working with the new setup for building Racket, and cut myself on many of it's rough edges. Some of these are about the package system in general, and some are about the new repository and build infrastructure. * When there's an installation error, it's really hard to tell what state you're in, or what you should do to fix it. Usually, the package manager just says that it's already installed. This is a general issue in the package system, and it's easily the most significant problem I encountered. The package system decides that something is installed very early, and many errors are possible after that. If the package didn't build, or it's dependencies didn't build, or any of many other things happened, the package system thinks everything is ok. The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job. Knowing whether 'raco setup' will succeed after those are in place isn't really possible beforehand and I don't think the idea of 'run raco setup and uninstall if it fails' is robust against the sorts of realistic and innocuous problems that raco setup can report. Maybe you could be more specific about what error you saw and what you wish had happened. * The error message when you look for a missing collection is really long if you have a lot of packages installed * Using parallel make, ie `make -j 8` doesn't work from the new top-level Makefile. * Even though unix systems don't use them, the extra libraries are downloaded and packaged up on unix platforms while building. * When installing a package with a long chain of dependencies, this message gets printed a lot: The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: at-exp-lib syntax-color The following packages are listed as dependencies, but are not currently installed, so they will be automatically installed: parser-tools and so on ... We should just collect all the packages to be installed and print them all at once (and perhaps prompt, the way `apt-get` does). Since package dependencies are in the package and not otherwise available, the manager is asking for permission to do a download. After doing the download, it can then look at the dependencies and see if there more. 'apt-get', etc have more monolithic repository structures than our package system, so they can analyze everything locally and compute the transitive closure. Furthermore, 'raco pkg' is prompting you, but you have it set on auto (the default) so it's equivalent to 'yes | apt-get' * I think it's possible to confuse the local catalog by moving directories around in the `pkg` directory. * `make local-catalog` is not incremental -- it always re-downloads and re-packs the external libraries. * Sometimes, errors can be hard to trace to a particular package. For example, this one was because there was a mistake in the future-visualizer package info.rkt file. But I had to go groveling in /var/tmp/ to figure that out. /var/tmp/pkg13711534991371153499937/info.rkt::212: read: expected a closing '' context...: /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:176:17: body /home/samth/tmp/pkg-racket/racket/lib/collects/syntax/module-reader.rkt:173:2: wrap-internal lang:read /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:50:8: temp32 /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:37:0: get-info/full/ext /home/samth/tmp/pkg-racket/racket/lib/collects/setup/getinfo.rkt:33:0 /home/samth/tmp/pkg-racket/racket/lib/collects/pkg/lib.rkt:1274:0: pkg-single-collection85 Are there other examples? This stack trace is useful to just put a with-handlers at the right spot. Other errors are also hard to track down. In general, it would be very helpful to catch setup errors and report them based on the package being installed and/or the collection being set up. Sam _ Racket Developers list: http://lists.racket-lang.org/dev -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list:
Re: [racket-dev] experience using the `pkg` branch
Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Robby _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
It does run 'raco setup', it just doesn't have much to do in response to a failure, at least right now. Carl Eastlund On Thu, Jun 13, 2013 at 11:14 PM, Robby Findler ro...@eecs.northwestern.edu wrote: Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Robby _ Racket Developers list: http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experience using the `pkg` branch
I know it runs it. I don't know why Jay writes The package system says something is installed when the files are in place and the link is made. From some perspective, that's its job.. I can't tell if there's some technical piece I'm missing or not (on the surface, these words sound almost lazy but I *KNOW* Jay is anything but lazy!) Robby On Thu, Jun 13, 2013 at 10:21 PM, Carl Eastlund c...@ccs.neu.edu wrote: It does run 'raco setup', it just doesn't have much to do in response to a failure, at least right now. Carl Eastlund On Thu, Jun 13, 2013 at 11:14 PM, Robby Findler ro...@eecs.northwestern.edu wrote: Also, Jay: can you explain more why 'raco setup' isn't something that we should think about as running inside the pkg manager? (I'm not saying that automatically rolling back packages is the right thing to do or anything like that, but I would like to understand the model you have better.) Robby _ Racket Developers list: http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
