Re: [racket-dev] sudo make install
At Wed, 12 Jan 2011 13:57:06 -0700, Jon Rafkind wrote: On 01/03/2011 02:11 PM, Matthew Flatt wrote: At Mon, 3 Jan 2011 14:55:05 -0500, Sam Tobin-Hochstadt wrote: On Mon, Jan 3, 2011 at 2:51 PM, Robby Findler ro...@eecs.northwestern.edu wrote: Or maybe there is another possible solution that involves changing how the distribution and or the docs build works? I think the conventional solution is for `make install' to do much less work than it current does. In normal Make-built software, `make' does all of the compilation, and `make install' just does the copying. That way, `make install' wouldn't be doing things like reading preferences and creating the '.racket' directory. Well, moving work from `make install' to `make' does nothing in itself. What you're suggesting is that we change what `make install' produces. In particular, I think you're suggesting that no user-specific actions take place as a result of a build and install. That change sounds ok to me, and it can be accomplished by passing `--no-user' to `raco setup' during `make install'. Unless anyone objects, I'll push that change soon. Was this change pushed? I did a simple search but didn't find it. Not until now. I kept looking for our old rationale, but I couldn't find it and I've given up. The no-user-setup-on-install default is controlled by `configure', and there's a new `--enable-usersetup' flag to get the old behavior. _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] raco setup hangs
This was a bug in the optimizer's tracking of inlining so that it tried to unroll `loop' forever. It's now fixed. At Mon, 10 Jan 2011 17:16:45 -0700, Jon Rafkind wrote: `raco setup' hangs when it tries to build my planet package. I narrowed down the problem to this file #lang racket/base (require racket/class) (define bar (mixin () () (define (foo) (let loop ([x 0]) (printf loop!\n) (loop (begin ;; uncomment for this to fail #; (send this anything) 2)) Put that in bar.rkt or whatever and add it as the main file of info.ss. If the line (send this anything) is uncommented then `raco setup' will hang. It dosen't print loop! or anything so its not obvious whats wrong. `raco make bar.rkt' works though. I'm not exactly sure when this behavior started occurring but it was something like 3 weeks ago. _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] raco setup hangs
thanks, it works now On 01/13/2011 01:14 PM, Matthew Flatt wrote: This was a bug in the optimizer's tracking of inlining so that it tried to unroll `loop' forever. It's now fixed. At Mon, 10 Jan 2011 17:16:45 -0700, Jon Rafkind wrote: `raco setup' hangs when it tries to build my planet package. I narrowed down the problem to this file #lang racket/base (require racket/class) (define bar (mixin () () (define (foo) (let loop ([x 0]) (printf loop!\n) (loop (begin ;; uncomment for this to fail #; (send this anything) 2)) Put that in bar.rkt or whatever and add it as the main file of info.ss. If the line (send this anything) is uncommented then `raco setup' will hang. It dosen't print loop! or anything so its not obvious whats wrong. `raco make bar.rkt' works though. I'm not exactly sure when this behavior started occurring but it was something like 3 weeks ago. _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
[racket-dev] atomic file update by write rename... not!
A common trick to atomically update a file x on a Unix filesystem is to write a file with a temporary name like tmp-x-7687564 (making up a fresh name each time) in the same directory as x and then use `rename-or-file-directory' to move tmp-x-7687564 to x. Unfortunately, this trick does not work for Windows. Don't use it for anything that needs to be portable. Here's why the trick works for Unix, at least when there's a single writer and multiple readers: If any process has x open while it is being replaced, the process continues to read from the old file, even though the file doesn't correspond to the file that is named x on the filesystem. The original file continues to exist until it is closed by all readers, while new readers of x get the new file. Furthermore, Unix guarantees that the rename is atomic; there's no time between x ceasing to refer to the old file and starting to refer to the new file. There are two problems with this approach under Windows. First, file rename is not guaranteed to be atomic (although it does seem to be atomic enough in everything I've tried, for whatever that's worth). Second, and more significantly in practice, while a file is opened for reading, you cannot use its name for a new file. You can delete a file while it's being read, but not only does the deleted file continue to exist, the delete file's *name* remains occupied until all uses of the file are closed. So, if x is opened by any readers, you cannot rename tmp-x-7687564 to x --- not even if you use the flag on the Win32 call to say that it's ok to replace the target file. Oddly enough, you're allowed to rename a Windows file while it is open. You can even delete the renamed file and then use the old name for a new file, even if the original file remains open by a reader who accessed it by the original name. Unfortunately, that possibility doesn't help to implement atomic update; if you follow a move x to old-x-7687564 with a move of tmp-x-7687564 to x, then for a significant period of time, there would be no x file. (BTW, I'm not clear on which constraints and behaviors are technically Windows and which are technically NTFS, but it hardly matters.) The solution? I haven't been able to find a solution that lets you just replace `rename-file-or-directory' with something else. The relatively new support for transactions in NTFS doesn't help unless all I/O is converted to use transactions, and transaction failures must be handled explicitly; it's way too painful. The only solutions I can find involve restructuring programs to add some new kind of synchronization. To support a parallel build, for example, Kevin had to change `raco setup' so that multiple processes do not try to compile the same file. (As it turns out, there were reasons to make the change independent of problems with atomic update, but the atomic-update problem finally forced us to make the change.) We haven't yet fixed the preferences file. I'll post a separate message on that one. Is there anything else in the main distribution that is uses `rename-file-or-directory' for atomic update? _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] atomic file update by write rename... not!
15 minutes ago, Matthew Flatt wrote: Is there anything else in the main distribution that is uses `rename-file-or-directory' for atomic update? Is the handin-server's use kosher? There are three uses of it, and the only one that looks suspicious is the last one, where ATTEMPT is renamed to SUCCESS-0. -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] atomic file update by write rename... not!
At Thu, 13 Jan 2011 16:30:08 -0500, Eli Barzilay wrote: 15 minutes ago, Matthew Flatt wrote: Is there anything else in the main distribution that is uses `rename-file-or-directory' for atomic update? Is the handin-server's use kosher? There are three uses of it, and the only one that looks suspicious is the last one, where ATTEMPT is renamed to SUCCESS-0. Those are directories, and you can't move to an existing directory name on any platform, right? I think the handin server probably depends on non-concurrent handins for a given account, though, since it shuffles many files and directories around. Probably it should use some sort of lock to enforce non-concurrency. _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] atomic file update by write rename... not!
On Thu, Jan 13, 2011 at 3:40 PM, Matthew Flatt mfl...@cs.utah.edu wrote: At Thu, 13 Jan 2011 16:30:08 -0500, Eli Barzilay wrote: 15 minutes ago, Matthew Flatt wrote: Is there anything else in the main distribution that is uses `rename-file-or-directory' for atomic update? Is the handin-server's use kosher? There are three uses of it, and the only one that looks suspicious is the last one, where ATTEMPT is renamed to SUCCESS-0. Those are directories, and you can't move to an existing directory name on any platform, right? I think the handin server probably depends on non-concurrent handins for a given account, though, since it shuffles many files and directories around. Probably it should use some sort of lock to enforce non-concurrency. In general, it seems like the handin server can safely make the assumption that there is only one sever running for a given directory and thus can do all of the syncronization at the Racket level (unlike the preferences file). Hopefully the preferences file is the only place (and things like raco setup where we know we're creating lots of processes). Robby _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
[racket-dev] the preferences file under Windows
Currently, the `get-preference' and `put-preferences' functions from `racket/file' (which are used by DrRacket and other Racket tools) use the write-to-temp-file-and-rename approach to atomic file update. There's a lock to keep multiple writers from trying to update the file, but no lock for readers. As explained in my previous message, that doesn't work for Windows. The writer lock is implemented by the existence of a lock file. That is, a writer cannot proceed unless it is able to create the lock file, and if some other process has created the lock file, then creation of the lock file by other processes fails (so they know that the file is locked). That much seems fine for Windows. The problem under Windows is to prevent readers from interfering with a writer --- since having a file open to read interferes with update --- the same as other writers. Option 1: Readers could take the lock on the preferences file in the same way as writers. The drawback of this option is that readers need write access to the directory containing the preferences file, since the lock is implemented by creating a file. Does that matter? It sounds wrong to me, but maybe it's just fine. Option 2: Use Windows file locking, and pick a new name for the preferences file under Windows. It takes a few steps to explain the new-name part... Not coincidentally, I recently added `port-try-file-lock?' and `port-file-unlock' to Racket, so OS-supported file locking is an option. (An evt form for locks would be nice, but OS APIs for file locking make that difficult or impossible.) Unfortunately, you can only hold the lock on a file if the file is open --- and having the preferences file open is the root of the problem for moving a new file into place. You can write to a file while holding a lock, of course, but writing directly to the data file risks losing information if the write is interrupted. We could solve this sub-problem by having two files: the first holds the preferences content, and the second exists only to be locked and unlocked. That way, holding a lock doesn't interfere with replacing the data file. Unfortunately (again), the lock file has to exist alongside the data file, and our existing preferences files are not accompanied by lock files. It's no good assuming that you don't need the lock if there's no lock file present, because the lock file might get created in between the time that you try to use the lock file and the time that you try to open the preferences file. That's why this option includes picking a new name for the preferences file (and its accompanying lock file) under Windows. The existing preferences file would be used as a read-only fallback when no file exists with the new preferences-file name. Any opinions? Or does anyone see other options? _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] the preferences file under Windows
30 minutes ago, Matthew Flatt wrote: Unfortunately (again), the lock file has to exist alongside the data file, and our existing preferences files are not accompanied by lock files. It's no good assuming that you don't need the lock if there's no lock file present, because the lock file might get created in between the time that you try to use the lock file and the time that you try to open the preferences file. Why not always use such a lock file, creating it if it's not there -- and then you can open it once per process, and lock/unlock it for each read/write of the actual file. Does this fail somehow? -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] the preferences file under Windows
At Thu, 13 Jan 2011 17:29:15 -0500, Eli Barzilay wrote: 30 minutes ago, Matthew Flatt wrote: Unfortunately (again), the lock file has to exist alongside the data file, and our existing preferences files are not accompanied by lock files. It's no good assuming that you don't need the lock if there's no lock file present, because the lock file might get created in between the time that you try to use the lock file and the time that you try to open the preferences file. Why not always use such a lock file, creating it if it's not there -- and then you can open it once per process, and lock/unlock it for each read/write of the actual file. Does this fail somehow? That's a good idea. It's a little bit of option 1, in that a reader will sometimes need to write a file --- but only in the transitional case of dealing with an existing preference file without a lock file. _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] the preferences file under Windows
Two minutes ago, Matthew Flatt wrote: At Thu, 13 Jan 2011 17:29:15 -0500, Eli Barzilay wrote: 30 minutes ago, Matthew Flatt wrote: Unfortunately (again), the lock file has to exist alongside the data file, and our existing preferences files are not accompanied by lock files. It's no good assuming that you don't need the lock if there's no lock file present, because the lock file might get created in between the time that you try to use the lock file and the time that you try to open the preferences file. Why not always use such a lock file, creating it if it's not there -- and then you can open it once per process, and lock/unlock it for each read/write of the actual file. Does this fail somehow? That's a good idea. It's a little bit of option 1, in that a reader will sometimes need to write a file --- but only in the transitional case of dealing with an existing preference file without a lock file. Yeah... If it wasn't for the transition, it would have already been there (created when the prefs file is created). (Another point is where the lock file is located, like putting it in some /dev/shm-like place so the file itself becomes just a way to name a kernel lock. I think that I've seen that discussed in some linux context, with a similar sounding locking scheme (where you open the file and then lock it).) -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] the preferences file under Windows
On Thu, Jan 13, 2011 at 4:36 PM, Matthew Flatt mfl...@cs.utah.edu wrote: At Thu, 13 Jan 2011 17:29:15 -0500, Eli Barzilay wrote: 30 minutes ago, Matthew Flatt wrote: Unfortunately (again), the lock file has to exist alongside the data file, and our existing preferences files are not accompanied by lock files. It's no good assuming that you don't need the lock if there's no lock file present, because the lock file might get created in between the time that you try to use the lock file and the time that you try to open the preferences file. Why not always use such a lock file, creating it if it's not there -- and then you can open it once per process, and lock/unlock it for each read/write of the actual file. Does this fail somehow? That's a good idea. It's a little bit of option 1, in that a reader will sometimes need to write a file --- but only in the transitional case of dealing with an existing preference file without a lock file. Also, I think we would have had to do that regardless, unless we wanted the installation process to create the lockfile or something. Robby _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
Re: [racket-dev] the preferences file under Windows
On Thu, Jan 13, 2011 at 4:42 PM, Eli Barzilay e...@barzilay.org wrote: Two minutes ago, Matthew Flatt wrote: At Thu, 13 Jan 2011 17:29:15 -0500, Eli Barzilay wrote: 30 minutes ago, Matthew Flatt wrote: Unfortunately (again), the lock file has to exist alongside the data file, and our existing preferences files are not accompanied by lock files. It's no good assuming that you don't need the lock if there's no lock file present, because the lock file might get created in between the time that you try to use the lock file and the time that you try to open the preferences file. Why not always use such a lock file, creating it if it's not there -- and then you can open it once per process, and lock/unlock it for each read/write of the actual file. Does this fail somehow? That's a good idea. It's a little bit of option 1, in that a reader will sometimes need to write a file --- but only in the transitional case of dealing with an existing preference file without a lock file. Yeah... If it wasn't for the transition, it would have already been there (created when the prefs file is created). (Another point is where the lock file is located, like putting it in some /dev/shm-like place so the file itself becomes just a way to name a kernel lock. I think that I've seen that discussed in some linux context, with a similar sounding locking scheme (where you open the file and then lock it).) Don't we only need this under windows? Robby _ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
