[julia-users] Running multiple scripts in parallel Julia sessions
I think GNU parallel is the best tool for that purpose. http://www.gnu.org/software/parallel/ You can pass -j option to control the number of maximum jobs at a time.
Re: [julia-users] Running multiple scripts in parallel Julia sessions
You could write your own pmap-like function (see its source code) that starts a new worker and then shuts it down again when done. Or, just wrap all your scripts inside modules? module Script1 # code that was in script1.jl end --Tim On Sunday, January 24, 2016 01:13:21 PM Ritchie Lee wrote: > The scripts contain a lot of global consts and other things. pmap seems to > mix the namespaces for all scripts executed on the same processor. > > On Sunday, January 24, 2016 at 12:06:05 AM UTC-8, Tim Holy wrote: > > pmap? > > > > --Tim > > > > On Saturday, January 23, 2016 07:29:07 PM Ritchie Lee wrote: > > > Do you mean using @spawn, success, or just from the command prompt? > > > > > > The scripts are long-running experiments where I am also tracking things > > > like CPU time. I would like them queued and run 4 at a time, i.e., if 1 > > > finishes ahead of others, then the next one will start running on the > > > > free > > > > > processor. Is there a way to schedule into separate processes? > > > > > > On Saturday, January 23, 2016 at 12:00:35 PM UTC-8, Stefan Karpinski > > > > wrote: > > > > Any reason not to run them all as separate processes? > > > > > > > > On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee > > > > > > > > wrote: > > > >> Let's say I have 10 julia scripts, scripts = ["script1.jl", > > > > "script2.jl", > > > > > >> , "script10.jl"] and I would like to run them in parallel in > > > > separate > > > > > >> Julia sessions, but 4 at a time (since I only have 4 cores on my > > > >> machine). > > > >> Is there any way to do this programmatically? > > > >> > > > >> I tried doing this: > > > >> > > > >> addprocs(4) > > > >> pmap(include, scripts) > > > >> > > > >> or > > > >> > > > >> addprocs(4) > > > >> @parallel for s in scripts > > > >> include(s) > > > >> end > > > >> > > > >> However, this seems to reuse the same session, so all the global > > > > consts > > > > > >> in the script file are colliding. I would like to make sure that the > > > >> namespaces are completely separate. > > > >> > > > >> Thanks!
Re: [julia-users] Running multiple scripts in parallel Julia sessions
The scripts contain a lot of global consts and other things. pmap seems to mix the namespaces for all scripts executed on the same processor. On Sunday, January 24, 2016 at 12:06:05 AM UTC-8, Tim Holy wrote: > > pmap? > > --Tim > > On Saturday, January 23, 2016 07:29:07 PM Ritchie Lee wrote: > > Do you mean using @spawn, success, or just from the command prompt? > > > > The scripts are long-running experiments where I am also tracking things > > like CPU time. I would like them queued and run 4 at a time, i.e., if 1 > > finishes ahead of others, then the next one will start running on the > free > > processor. Is there a way to schedule into separate processes? > > > > On Saturday, January 23, 2016 at 12:00:35 PM UTC-8, Stefan Karpinski > wrote: > > > Any reason not to run them all as separate processes? > > > > > > On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee > > > > > > wrote: > > >> Let's say I have 10 julia scripts, scripts = ["script1.jl", > "script2.jl", > > >> , "script10.jl"] and I would like to run them in parallel in > separate > > >> Julia sessions, but 4 at a time (since I only have 4 cores on my > > >> machine). > > >> Is there any way to do this programmatically? > > >> > > >> I tried doing this: > > >> > > >> addprocs(4) > > >> pmap(include, scripts) > > >> > > >> or > > >> > > >> addprocs(4) > > >> @parallel for s in scripts > > >> include(s) > > >> end > > >> > > >> However, this seems to reuse the same session, so all the global > consts > > >> in the script file are colliding. I would like to make sure that the > > >> namespaces are completely separate. > > >> > > >> Thanks! > >
Re: [julia-users] Running multiple scripts in parallel Julia sessions
pmap? --Tim On Saturday, January 23, 2016 07:29:07 PM Ritchie Lee wrote: > Do you mean using @spawn, success, or just from the command prompt? > > The scripts are long-running experiments where I am also tracking things > like CPU time. I would like them queued and run 4 at a time, i.e., if 1 > finishes ahead of others, then the next one will start running on the free > processor. Is there a way to schedule into separate processes? > > On Saturday, January 23, 2016 at 12:00:35 PM UTC-8, Stefan Karpinski wrote: > > Any reason not to run them all as separate processes? > > > > On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee > > > > wrote: > >> Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl", > >> , "script10.jl"] and I would like to run them in parallel in separate > >> Julia sessions, but 4 at a time (since I only have 4 cores on my > >> machine). > >> Is there any way to do this programmatically? > >> > >> I tried doing this: > >> > >> addprocs(4) > >> pmap(include, scripts) > >> > >> or > >> > >> addprocs(4) > >> @parallel for s in scripts > >> include(s) > >> end > >> > >> However, this seems to reuse the same session, so all the global consts > >> in the script file are colliding. I would like to make sure that the > >> namespaces are completely separate. > >> > >> Thanks!
Re: [julia-users] Running multiple scripts in parallel Julia sessions
Do you mean using @spawn, success, or just from the command prompt? The scripts are long-running experiments where I am also tracking things like CPU time. I would like them queued and run 4 at a time, i.e., if 1 finishes ahead of others, then the next one will start running on the free processor. Is there a way to schedule into separate processes? On Saturday, January 23, 2016 at 12:00:35 PM UTC-8, Stefan Karpinski wrote: > > Any reason not to run them all as separate processes? > > On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee > wrote: > >> Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl", >> , "script10.jl"] and I would like to run them in parallel in separate >> Julia sessions, but 4 at a time (since I only have 4 cores on my machine). >> Is there any way to do this programmatically? >> >> I tried doing this: >> >> addprocs(4) >> pmap(include, scripts) >> >> or >> >> addprocs(4) >> @parallel for s in scripts >> include(s) >> end >> >> However, this seems to reuse the same session, so all the global consts >> in the script file are colliding. I would like to make sure that the >> namespaces are completely separate. >> >> Thanks! >> > >
Re: [julia-users] Running multiple scripts in parallel Julia sessions
Any reason not to run them all as separate processes? On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee wrote: > Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl", > , "script10.jl"] and I would like to run them in parallel in separate > Julia sessions, but 4 at a time (since I only have 4 cores on my machine). > Is there any way to do this programmatically? > > I tried doing this: > > addprocs(4) > pmap(include, scripts) > > or > > addprocs(4) > @parallel for s in scripts > include(s) > end > > However, this seems to reuse the same session, so all the global consts in > the script file are colliding. I would like to make sure that the > namespaces are completely separate. > > Thanks! >
[julia-users] Running multiple scripts in parallel Julia sessions
Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl", , "script10.jl"] and I would like to run them in parallel in separate Julia sessions, but 4 at a time (since I only have 4 cores on my machine). Is there any way to do this programmatically? I tried doing this: addprocs(4) pmap(include, scripts) or addprocs(4) @parallel for s in scripts include(s) end However, this seems to reuse the same session, so all the global consts in the script file are colliding. I would like to make sure that the namespaces are completely separate. Thanks!