I was wondering if anybody can point me towards a tutorial or a large code base using parallel computing. Everything that is discussed so far in the docs and books is super simple - take a function, run it in parallel, the end.
To explain, I'm working on a full stack MVC web framework - so think many functions and a few types grouped in maybe 20 modules. Plus more or less 20 other external modules. The workflow that I'm after is: 1. bootstrap - load the necessary components to start up the framework (parsing command line args, loading configuration, setting up include paths, etc) 2. start an instance of HttpServer and listen to a port 3. when the server receives a request it invokes a function of the Router module which is the entry point into the MVC stack 4. once the Router gets the request, it's pushed up the MVC stack and at the end a HttpServer.Response instance is returned That being said, a. my strategy is simple: for each request, spawn the function call at #3 to a worker b. imagine that what's at #4 represents 80% of the app, with a multitude of functions being invoked across a multitude of modules (ORM, controller, Models, Loggers, Databases, Caching, Sessions, Authentication, etc). Everything works great single process, but getting the stack to run on multiple workers it's a nightmare. I got it to the point where at #3 I can invoke a simple function call (something like returning a string or a date) and run it on multiple workers. But when I try to invoke the Router and snowball the framework, I end up in an avalanche of unknown references. The codebase is now littered with calls to @everywhere that add a lot of noise. But up to this point I still wasn't able to make it work. The errors come from deep within the Julia stack, the workers crash saying that a module or function can't be found but there's no stack trace to point me towards the location so it's really trial end error commenting and uncommenting "include" and "using" statements to see what gives, I get errors from within external modules, etc. I guess what I'm trying to say is that my experience with parallel computing applied to a large codebase is very frustrating. And I was wondering if anybody has done a parallel implementation in a larger codebase and has any tips on how to attack it. P.S. I might be spoiled by Elixir, but ideally I'd like to be able to spawn processes and functions whenever I need, and have the compiler take care of making the code available across workers. Another thing that would be very useful, also inspired by Elixir, is the supervisor design pattern, to look over and restart the workers when they crash.
