This is a random example of an error - not really sure how to debug this, seems to crash within the Postgres library. The dump is long but not really helpful.
12-Sep 21:39:38:WARNING:root:Module __anon__ not defined on process 5 12-Sep 21:39:38:WARNING:root:Module __anon__ not defined on process 3 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 2 fatal error on fatal error on 3: ERROR: attempt to send to unknown socket 4: ERROR: attempt to send to unknown socket 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 3 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 5 fatal error on 3: ERROR: attempt to send to unknown socket 12-Sep 21:39:39:WARNING:root:Module __anon__ not defined on process 3 fatal error on 2: ERROR: attempt to send to unknown socket ERROR: LoadError: On worker 2: LoadError: On worker 2: LoadError: LoadError: LoadError: LoadError: LoadError: ProcessExitedException() in yieldto at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in wait at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib (repeats 3 times) in take! at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib (repeats 2 times) in remotecall_fetch at multi.jl:745 in remotecall_fetch at multi.jl:750 in anonymous at multi.jl:1396 ...and 1 other exceptions. in include_string at loading.jl:282 in include_from_node1 at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in include_string at loading.jl:282 in include_from_node1 at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in eval at /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/Database.jl:1 in include_string at loading.jl:282 in include_from_node1 at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in include_string at loading.jl:282 in include_from_node1 at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in eval at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in anonymous at multi.jl:1394 in run_work_thunk at multi.jl:661 in remotecall_fetch at multi.jl:734 in remotecall_fetch at multi.jl:750 in anonymous at multi.jl:1396 while loading /Users/adrian/.julia/v0.4/PostgreSQL/src/../deps/build.jl, in expression starting on line 9 while loading /Users/adrian/.julia/v0.4/PostgreSQL/src/PostgreSQL.jl, in expression starting on line 8 while loading /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/database_adapters/PostgreSQLDatabaseAdapter.jl, in expression starting on line 3 while loading /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/Database.jl, in expression starting on line 7 while loading /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/commands.jl, in expression starting on line 11 in remotecall_fetch at multi.jl:735 in remotecall_fetch at multi.jl:750 in anonymous at multi.jl:1396 ...and 1 other exceptions. in sync_end at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in anonymous at multi.jl:1405 in include_string at loading.jl:282 in include_from_node1 at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in require at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in eval at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in anonymous at multi.jl:1394 in anonymous at multi.jl:923 in run_work_thunk at multi.jl:661 [inlined code] from multi.jl:923 in anonymous at task.jl:63 while loading /Users/adrian/Dropbox/Projects/jinnie/lib/Genie/src/Genie.jl, in expression starting on line 1 in remotecall_fetch at multi.jl:747 in remotecall_fetch at multi.jl:750 in anonymous at multi.jl:1396 ...and 4 other exceptions. in sync_end at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in anonymous at multi.jl:1405 in include at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in include_from_node1 at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in process_options at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib in _start at /usr/local/Cellar/julia/0.4.6_1/lib/julia/sys.dylib while loading /Users/adrian/Dropbox/Projects/jinnie/genie.jl, in expression starting on line 16 luni, 12 septembrie 2016, 23:01:51 UTC+2, Adrian Salceanu a scris: > > I was wondering if anybody can point me towards a tutorial or a large code > base using parallel computing. Everything that is discussed so far in the > docs and books is super simple - take a function, run it in parallel, the > end. > > To explain, I'm working on a full stack MVC web framework - so think many > functions and a few types grouped in maybe 20 modules. Plus more or less 20 > other external modules. The workflow that I'm after is: > > 1. bootstrap - load the necessary components to start up the framework > (parsing command line args, loading configuration, setting up include > paths, etc) > 2. start an instance of HttpServer and listen to a port > 3. when the server receives a request it invokes a function of the Router > module which is the entry point into the MVC stack > 4. once the Router gets the request, it's pushed up the MVC stack and at > the end a HttpServer.Response instance is returned > > That being said, > a. my strategy is simple: for each request, spawn the function call at #3 > to a worker > b. imagine that what's at #4 represents 80% of the app, with a multitude > of functions being invoked across a multitude of modules (ORM, controller, > Models, Loggers, Databases, Caching, Sessions, Authentication, etc). > > Everything works great single process, but getting the stack to run on > multiple workers it's a nightmare. I got it to the point where at #3 I can > invoke a simple function call (something like returning a string or a date) > and run it on multiple workers. But when I try to invoke the Router and > snowball the framework, I end up in an avalanche of unknown references. > > The codebase is now littered with calls to @everywhere that add a lot of > noise. But up to this point I still wasn't able to make it work. The errors > come from deep within the Julia stack, the workers crash saying that a > module or function can't be found but there's no stack trace to point me > towards the location so it's really trial end error commenting and > uncommenting "include" and "using" statements to see what gives, I get > errors from within external modules, etc. > > I guess what I'm trying to say is that my experience with parallel > computing applied to a large codebase is very frustrating. And I was > wondering if anybody has done a parallel implementation in a larger > codebase and has any tips on how to attack it. > > > P.S. > I might be spoiled by Elixir, but ideally I'd like to be able to spawn > processes and functions whenever I need, and have the compiler take care of > making the code available across workers. > Another thing that would be very useful, also inspired by Elixir, is the > supervisor design pattern, to look over and restart the workers when they > crash. >
