2011/10/24 Hiroshi Nakamura <nakah...@gmail.com>: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi, > > On 10/24/2011 11:03 PM, Charles Oliver Nutter wrote: >> Nahi planted an interesting seed on Twitter...what if we could >> parallelize parsing of Ruby files when loading a large >> application? > > I once stated this idea for CRuby in [ruby-list:48136], as a possible > feature in a discussion for 'require should take multiple features as > an array'. (The proposal is not mine) > > require ['csv', 'json', 'yaml'] > > I said 'it would make require parallelize' but Matz responded at > [ruby-list:48140] > > Would you like to take pains over the problem depending on the > loading timing throughout life? Give me a break!
Assuming I understand this, I agree that many people will keep reporting problems when it boils down the reporter lack of understanding of the issues involving the feature. The example earlier in the thread would routinely happen and people would report it as a big because they would not understand what was wrong. > > In addition, you might know he hates autoload [ruby-core:39844]. Sigh, > we don't work very well. > > Back to the topic. > >> At a naive level, parallelizing the parse of an individual file is >> tricky to impossible; the parser state is very much straight-line. >> But perhaps it's possible to parallelize loading of many files? >> >> I started playing with parallelizing calls to the parser, but that >> doesn't really help anything; every call to the parser blocks >> waiting for it to complete, and the contents are not interpreted >> until after that point. That means that "require" lines remain >> totally opaque, preventing us from proactively starting threaded >> parses of additional files. But there lies the opportunity: what if >> load/require requests were done as Futures, require/load lines were >> eagerly interpreted by submitting load/require requests to a thread >> pool, and child requires could be loading and parsing at the same >> time as the parent file...without conflicting. > > I might misunderstanding something. But you're discussing only 'load' > (read in source as a stream) and 'parse' (create AST), not evaluating > it, right? Or does it include evaluation phase? You are correct Charlies spike only constructs the AST in parallel. So no execution has happened yet. We just have all the data structures primed and ready to start executing. > > Say: > > a.rb > require 'a/b' > require 'a/c' > > a/b.rb > module A > sleep 10 > module B > FOO = 1 > end > end > > a/c.rb > module A > module C > BAR = B::FOO > end > end > > How does your Futures work? > > Ah, a new mail arrived from Jonathan Coveney. I think I'm saying the > same thing. > > Changing LoadService as you stated sounds it includes evaluation > phase. But I'm a little bit far from JRuby source code recently, I > might be wrong. > > // NaHi > >> In order to do this, I think we would need to make the following >> modifications: >> >> * LoadService would need to explose Future-based versions of >> "load" and "require". The initial file loaded as the "main" script >> would be synchronous, but subsequent requires and loads could be >> shunted to a thread pool. * The parser would need to initiate eager >> load+parser of files encountered in require-like and load-like >> lines. This load+parse would encompass filesystem searching plus >> content parsing, so all the heavy lifting of booting a file would >> be pushed into the thread pool. * Somewhere (perhaps in >> LoadService) we would maintain an LRU cache mapping from file paths >> to ASTs. The cache would contain Futures; getting the actual parsed >> library would then simply be a matter of Future.get, allowing many >> of the load+parses to be done asynchronously. >> >> For a system like Rails, where there might be hundreds of files >> loaded, this could definitely improve startup performance. >> >> Thoughts? >> >> - Charlie > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > > iQEcBAEBAgAGBQJOpkFjAAoJEC7N6P3yLbI2OEAIAMXfaOM666jS40UFm7uHY8/H > 2qpkWp8az9cnt8GvBfyx6d1h9aTeo8TQr/r8mXod/Ahq/WyagbbMIGRALr2AA8xA > sDOdw7X5s1QObF4CtlnQ5fRveOtwvPYy1cNgdSizetb2yx6R3xLbdnSeIYbYwsMQ > p5q3GxSnU4o8qY9hm0IzzU9t6qKuWsRXVgknLdJWfIFfccpe8z3HHDCCDScXC08p > nk70hA3px37tmxJM0GNSSNQs/V9D591wegE/xkO/jaM11mAggQPtHCIUxeSy/ikF > vOJ/9N2c79TZHffXsnfq6yEhyBIbWUudhwBt7exIkbmSpGBRVzlwfh2GPj9D5Ss= > =OdJs > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > -- blog: http://blog.enebo.com twitter: tom_enebo mail: tom.en...@gmail.com --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
