Re: [racket-dev] PLTCOMPILEDROOTS
At Tue, 9 Oct 2012 12:22:59 -0700, Dan Liebgold wrote: On Tue, Oct 9, 2012 at 11:32 AM, Matthew Flatt mfl...@cs.utah.edu wrote: At Tue, 9 Oct 2012 18:00:14 + (UTC), Dan Liebgold wrote: It still seems strange to me that we should implement our own filesystem of some sort to get decent filesystem performance. Indeed. I think in game development we tend to be a bit nuts about performance, but I do have some numbers to back it up! During a single startup of our app we trigger about 7000 file operations in the collects directory. When stored on a local hard drive it takes about 2 seconds. On a network drive that number climbs to about 30 seconds. And that is just loading the collects we utilize. A factor of 15 is a lot. On the good cs.utah.edu (my department) network filesystem, Racket starts in about the same as on my local machine. One the less-good eng.utah.edu (my college) network filesystem, starting Racket takes about 5 times as long, which is uncomfortable. People who know about network filesystems tell me that the difference is in how the filesystems are tuned. They think a single file to hold bytecode will likely help by trading meta-data operations on many files for many read operations on a single file. I guess we should try this, but I think we should keep it specific to bytecode loading. That is, I don't think we should try to splice a .zip-based filesystem into Racket's core filesystem operations. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] PLTCOMPILEDROOTS
On Wed, Oct 10, 2012 at 10:19 AM, Matthew Flatt mfl...@cs.utah.edu wrote: I guess we should try this, but I think we should keep it specific to bytecode loading. That is, I don't think we should try to splice a .zip-based filesystem into Racket's core filesystem operations. I would think that something along the lines of how classpaths work in Java would be the right direction -- ie, PLTCOLLECTS could include files (perhaps with a distinguished extension) in addition to directories, and those would be ZIP files whose internal directory structure was treated the way a directory entry directly in PLTCOLLECTS is treated. -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] PLTCOMPILEDROOTS
Just now, Matthew Flatt wrote: I guess we should try this, but I think we should keep it specific to bytecode loading. That is, I don't think we should try to splice a .zip-based filesystem into Racket's core filesystem operations. That's what I had in mind, though splicing a zip at the lower level is tempting for an even easier distribution of content... But it sounds better to avoid it and instead, when needed, go for the lowlevel hooks so that any arbitrary code could be plugged in to access files. [ BTW, a cute mini project would be to start with some bindings to FUSE, then have a single racket process that has code for implementing a FS and also use it -- so you end up having racket code that implements the access, and code that uses it. But I think that there's no per-process local mounting via fuse... ] -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] PLTCOMPILEDROOTS
On Wed, Oct 10, 2012 at 7:19 AM, Matthew Flatt mfl...@cs.utah.edu wrote: People who know about network filesystems tell me that the difference is in how the filesystems are tuned. They think a single file to hold bytecode will likely help by trading meta-data operations on many files for many read operations on a single file. That matches our experience precisely. -- Dan Liebgold[dan.liebg...@gmail.com] _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] PLTCOMPILEDROOTS
FWIW, a zip file may not be a good format because the file index is just a O(n) access list (no matter how many sub-directories the zip file contains.) I didn't find any standard archive formats with O(log n) or better performance, which is why DrDr uses a custom format: http://jeapostrophe.github.com/blog/2012/08/13/drdr-archive/ Jay On Wed, Oct 10, 2012 at 8:23 AM, Sam Tobin-Hochstadt sa...@ccs.neu.edu wrote: On Wed, Oct 10, 2012 at 10:19 AM, Matthew Flatt mfl...@cs.utah.edu wrote: I guess we should try this, but I think we should keep it specific to bytecode loading. That is, I don't think we should try to splice a .zip-based filesystem into Racket's core filesystem operations. I would think that something along the lines of how classpaths work in Java would be the right direction -- ie, PLTCOLLECTS could include files (perhaps with a distinguished extension) in addition to directories, and those would be ZIP files whose internal directory structure was treated the way a directory entry directly in PLTCOLLECTS is treated. -- sam th sa...@ccs.neu.edu _ Racket Developers list: http://lists.racket-lang.org/dev -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] PLTCOMPILEDROOTS
At Tue, 9 Oct 2012 18:00:14 + (UTC), Dan Liebgold wrote: Now is there a way to archive the bytecode files in general (thinking .jar files), and the standard collects in particular? Right now our app will stat and load thousands of .zo files during it's normal operation. In performance critical sections this is a lot of time wasted on file operations (often over a network.) There's nothing for that right now, although I did consider trying to solve that problem while working on PLTCOMPILEDROOTS. It still seems strange to me that we should implement our own filesystem of some sort to get decent filesystem performance. _ Racket Developers list: http://lists.racket-lang.org/dev
