Thanks - haven't thought of using Dagger (don't have previous experience with it either). I've read the docs now but I'm not sure how using it would help, maybe I'm missing something?
My problem is really about running deeply nested function calls across multiple modules, in parallel. Making the code available to the workers involves a lot of back tracking, module by module and function by function, prepending @everywhere, well... everywhere. Cherry picking function calls in a few thousands lines codebase to make it load across processes is a task for computers, not people. Is this the idiomatic Julia way, because it really feels like I'm misusing the language? Am I supposed to end up with tens of calls to @everywhere? - it doesn't feel right... luni, 12 septembrie 2016, 23:53:06 UTC+2, dnm a scris: > > Have you tried Dagger.jl <https://github.com/JuliaParallel/Dagger.jl> to > set up a DAG of computations you need performed? > > On Monday, September 12, 2016 at 5:04:26 PM UTC-4, Adrian Salceanu wrote: >> >> This is a random example of an error - not really sure how to debug this, >> seems to crash within the Postgres library. The dump is long but not really >> helpful. >> >> 12-Sep 21:39:38:WARNING:root:Module __anon__ not defined on process 5 >> 12-Sep 21:39:38:WARNING:root:Module __anon__ not defined on process 3 >> 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 2 >> fatal error on fatal error on 3: ERROR: attempt to send to unknown socket >> 4: ERROR: attempt to send to unknown socket >> 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 3 >> 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 5 >> fatal error on 3: ERROR: attempt to send to unknown socket >> 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 3 >> fatal error on 2: ERROR: attempt to send to unknown socket >> ERROR: LoadError: On worker 2: >> LoadError: On worker 2: >> LoadError: LoadError: LoadError: LoadError: LoadError: >> ProcessExitedException() >> in yieldto at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in wait at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib (repeats >> 3 times) >> in take! at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib (repeats >> 2 times) >> in remotecall_fetch at multi.jl:745 >> in remotecall_fetch at multi.jl:750 >> in anonymous at multi.jl:1396 >> >> ...and 1 other exceptions. >> >> in include_string at loading.jl:282 >> in include_from_node1 at >> /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in include_string at loading.jl:282 >> in include_from_node1 at >> /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in eval at >> /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/Database.jl:1 >> in include_string at loading.jl:282 >> in include_from_node1 at >> /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in include_string at loading.jl:282 >> in include_from_node1 at >> /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in eval at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in anonymous at multi.jl:1394 >> in run_work_thunk at multi.jl:661 >> in remotecall_fetch at multi.jl:734 >> in remotecall_fetch at multi.jl:750 >> in anonymous at multi.jl:1396 >> while loading /Users/adrian/.julia/v0.4/PostgreSQL/src/../deps/build.jl, >> in expression starting on line 9 >> while loading /Users/adrian/.julia/v0.4/PostgreSQL/src/PostgreSQL.jl, in >> expression starting on line 8 >> while loading >> /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/database_adapters/PostgreSQLDatabaseAdapter.jl, >> >> in expression starting on line 3 >> while loading >> /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/Database.jl, in >> expression starting on line 7 >> while loading >> /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/commands.jl, in >> expression starting on line 11 >> in remotecall_fetch at multi.jl:735 >> in remotecall_fetch at multi.jl:750 >> in anonymous at multi.jl:1396 >> >> ...and 1 other exceptions. >> >> in sync_end at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in anonymous at multi.jl:1405 >> in include_string at loading.jl:282 >> in include_from_node1 at >> /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in eval at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in anonymous at multi.jl:1394 >> in anonymous at multi.jl:923 >> in run_work_thunk at multi.jl:661 >> [inlined code] from multi.jl:923 >> in anonymous at task.jl:63 >> while loading >> /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/Genie.jl, in expression >> starting on line 1 >> in remotecall_fetch at multi.jl:747 >> in remotecall_fetch at multi.jl:750 >> in anonymous at multi.jl:1396 >> >> ...and 4 other exceptions. >> >> in sync_end at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in anonymous at multi.jl:1405 >> in include at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in include_from_node1 at >> /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in process_options at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> in _start at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib >> while loading /Users/adrian/Dropbox/Projects/jinnie/genie.jl, in >> expression starting on line 16 >> >> luni, 12 septembrie 2016, 23:01:51 UTC+2, Adrian Salceanu a scris: >>> >>> I was wondering if anybody can point me towards a tutorial or a large >>> code base using parallel computing. Everything that is discussed so far in >>> the docs and books is super simple - take a function, run it in parallel, >>> the end. >>> >>> To explain, I'm working on a full stack MVC web framework - so think >>> many functions and a few types grouped in maybe 20 modules. Plus more or >>> less 20 other external modules. The workflow that I'm after is: >>> >>> 1. bootstrap - load the necessary components to start up the framework >>> (parsing command line args, loading configuration, setting up include >>> paths, etc) >>> 2. start an instance of HttpServer and listen to a port >>> 3. when the server receives a request it invokes a function of the >>> Router module which is the entry point into the MVC stack >>> 4. once the Router gets the request, it's pushed up the MVC stack and at >>> the end a HttpServer.Response instance is returned >>> >>> That being said, >>> a. my strategy is simple: for each request, spawn the function call at >>> #3 to a worker >>> b. imagine that what's at #4 represents 80% of the app, with a multitude >>> of functions being invoked across a multitude of modules (ORM, controller, >>> Models, Loggers, Databases, Caching, Sessions, Authentication, etc). >>> >>> Everything works great single process, but getting the stack to run on >>> multiple workers it's a nightmare. I got it to the point where at #3 I can >>> invoke a simple function call (something like returning a string or a date) >>> and run it on multiple workers. But when I try to invoke the Router and >>> snowball the framework, I end up in an avalanche of unknown references. >>> >>> The codebase is now littered with calls to @everywhere that add a lot of >>> noise. But up to this point I still wasn't able to make it work. The errors >>> come from deep within the Julia stack, the workers crash saying that a >>> module or function can't be found but there's no stack trace to point me >>> towards the location so it's really trial end error commenting and >>> uncommenting "include" and "using" statements to see what gives, I get >>> errors from within external modules, etc. >>> >>> I guess what I'm trying to say is that my experience with parallel >>> computing applied to a large codebase is very frustrating. And I was >>> wondering if anybody has done a parallel implementation in a larger >>> codebase and has any tips on how to attack it. >>> >>> >>> P.S. >>> I might be spoiled by Elixir, but ideally I'd like to be able to spawn >>> processes and functions whenever I need, and have the compiler take care of >>> making the code available across workers. >>> Another thing that would be very useful, also inspired by Elixir, is the >>> supervisor design pattern, to look over and restart the workers when they >>> crash. >>> >>
