Hi all, a discussion on the IRC channel prompts me to ask. >From a design point of view why does `using` behave the way it does to remote workers?
Assume some module called `FooModule` which exports `bar` `using FooModule` locally loads the module, and brings it into scope -- so running locally `bar` works, as does running locally `FooModule.bar` on all workers it loads the module, but does not bring it into scope so running on worker 2 (for example) `bar` give an unknown function error, but running `FooModule.bar` works `@everywhere using FooModule` causes it to be loaded locally and thus on all workers, and also to be loaded on all workers and brought into scope. Thus you get warnings about replacing modules as is loaded twice, but everything is in scope, so running on worker 2 (again for example) `bar` works. Why have this behavior? Wouldn't it be cleaner is using only acted locally? Discussion log from IRC follows for further context: Regards Frames > Day changed to 26 Nov 2015 > 05:37 < Travisty> Is there an easy way to use a single file to load code > for > multiple workers /and/ act as a driver script? Currently > I > have been putting an “@everywhere begin … end” block in > the > script to make sure that all the workers have the > appropriate > functions defined. > 05:38 < Travisty> But when I say “using SomeModule” inside the @everywhere > block, I get a bunch of warnings about the module being > replaced, which makes me think this might not be the > desired > approach > 08:39 < Frames> @Travisty it took me quiet a while to work out how to do > using > everywhere > 08:40 < Frames> So let me explain, that the knowledge may be shared > 08:40 < Travisty> Frames: sure, thanks > 08:40 < Travisty> (I should mention that what I’ve done works, but it > gives the > annoying warning about the modules being replaced) > 08:42 < Frames> If you run `using FooModule` on the main window, then > `FooModule` is loaded both locally and in all workers, and > furthermore locally (only) all exported functions are > brought > into scope. (continued in next messagee...) > 08:43 < Frames> So if `FooModule` exports `bar`, locally you can run > `bar`, but > on all workers you much run `FooModule.bar` (which also > works > locally) > 08:47 < Frames> On the other hand if you run `@everywhere using > `FooModule.bar`, then the stuff that happens when you run > `using FooModule` happens (as it runs locally and on all > workers), but also Simultainiously *!* it tiggers `using > FooModule` as a local command on the remote workers bring > it > into scope so now `bar` can run everywhere. (But this > does not > trigger all other remote workers to again get the remote > load > but > 08:47 < Frames> do not bring into scope, as under the default cluster > manager > topology workers can't see each other) > 08:49 < Frames> The inportant footnote *!* is that this can trigger a > (mostly > harmless) Race condition with the modules all trying to > precompile and cache the module at the same time. This > breaks > some modules (yet to work out which or exactly what goes > on) > 08:51 < Travisty> Frames: Yeah, I read about the behaviour of using > 08:51 < Travisty> is there an argument for it behaving this way? > 08:51 < Travisty> I mean, importing the module on the workers but not > bringing > the exported names into the namespace? > 08:52 < Travisty> Is there a way to get the workers to also import the > namespace of the module? > 08:52 < Frames> The best way I have for doing this is to first run `import > FooModule` then `@everywhere using FooModule`, which > causes it > to be loaded locally, then it is not recompiled again as > it is > already compiled locally, then `@everywhere using > FooModule` > brings it into scope everywhere. Thus you get no warnings > as it > has Not being loaded, as using doesn't trigger locally thus > does not trigger the remote loads in the > 08:52 < Frames> workers (WWhich may itself be a bug_ > 08:52 < Frames> ) > 08:53 < Travisty> ah > 08:53 < Travisty> so import on the driver and then @everywhere using > 08:53 < Frames> --- I am really not sure why it is that way. > 08:53 < Frames> ^Yes. Import then @everwhere using. > 08:54 < Frames> I was trying to make a macro to do it, but it does not > seem to > like. > 08:54 < Frames> Shall I copy and post this whole discussion to the mailing > list? > 08:54 < Travisty> Maybe a summary would be better > 08:55 < Frames> I think a summary, with the original discussion quoted > below > for more context. > 08:55 < Travisty> Couldn’t you just define a macro @pusing ModuleName that > expands to “include ModuleName; @everwhere using > ModuleName"? > 08:55 < Frames> include? You mean import right? > 08:55 < Travisty> yeah, right > 08:56 < Frames> THat was my idea for the macro yes, but apparently import > can't > be run form inside a "function" which apparently is what it > thinks a macro is at some point. >
