Nahi planted an interesting seed on Twitter...what if we could
parallelize parsing of Ruby files when loading a large application?
At a naive level, parallelizing the parse of an individual file is
tricky to impossible; the parser state is very much straight-line. But
perhaps it's possible to parallelize loading of many files?
I started playing with parallelizing calls to the parser, but that
doesn't really help anything; every call to the parser blocks waiting
for it to complete, and the contents are not interpreted until after
that point. That means that "require" lines remain totally opaque,
preventing us from proactively starting threaded parses of additional
files. But there lies the opportunity: what if load/require requests
were done as Futures, require/load lines were eagerly interpreted by
submitting load/require requests to a thread pool, and child requires
could be loading and parsing at the same time as the parent
file...without conflicting.
In order to do this, I think we would need to make the following modifications:
* LoadService would need to explose Future-based versions of "load"
and "require". The initial file loaded as the "main" script would be
synchronous, but subsequent requires and loads could be shunted to a
thread pool.
* The parser would need to initiate eager load+parser of files
encountered in require-like and load-like lines. This load+parse would
encompass filesystem searching plus content parsing, so all the heavy
lifting of booting a file would be pushed into the thread pool.
* Somewhere (perhaps in LoadService) we would maintain an LRU cache
mapping from file paths to ASTs. The cache would contain Futures;
getting the actual parsed library would then simply be a matter of
Future.get, allowing many of the load+parses to be done
asynchronously.
For a system like Rails, where there might be hundreds of files
loaded, this could definitely improve startup performance.
Thoughts?
- Charlie
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
