The shmem stuff in PTools.jl is quite old. Do check out the current master which has support for SharedArrays - see http://docs.julialang.org/en/latest/manual/parallel-computing/#shared-arrays-experimental-unix-only-feature- which is definitely more usable.
On Thu, Feb 13, 2014 at 5:00 AM, Jim Christoff <[email protected]>wrote: > I'm a little confused as to what functions are getting executed in your > example. Is it mypid()?. Could you point out what is the function. > > > On Saturday, May 4, 2013 1:57:18 AM UTC-4, Amit Murthy wrote: >> >> Folks, >> >> Please do checkout PTools <https://github.com/amitmurthy/PTools.jl> . I >> hope to grow it into a collection of utilities for parallel computation. >> >> Currently the following are available. >> >> - >> >> ServerTasks - These are long running tasks that simply processes >> incoming requests in a loop. Useful in situations where state needs to be >> maintained across function calls. State can be maintained and retrieved >> using the task_local_storage methods. >> - >> >> SharedMemory - Useful in the event of parallel processing on a single >> large multi-core machine. Avoids the overhead associated with >> sending/recieving large data sets. >> >> <https://github.com/amitmurthy/PTools.jl#usage>Usage >> >> Typical usage pattern will be >> >> - >> >> start_stasks - Start Server Tasks, optionally with shared memory >> mappings. >> - >> >> Execute a series of functions in parallel on these tasks using >> multiple invocations of pmap_stasks >> - >> >> SomeFunction >> - >> >> SomeOtherFunction >> - >> >> SomeOtherFunction . . . >> - >> >> stop_stasks - Stop all Server Tasks and free shared memory if >> required. >> >> The user specified functions in pmap_stasks can store and retrieve state >> information using the task_local_storage functions. >> <https://github.com/amitmurthy/PTools.jl#example>Example >> >> The best way to understand what is available is by example: >> >> - specify shared memory configuration. >> >> using PTools >> >> shmcfg = [ShmCfg(:svar1, Int32, 64*1024), ShmCfg(:svar2, Uint8, (100,100))] >> >> >> - >> >> the above line requests for a 64K Int32 array bound to svar1, and a >> 100x100 Uint8 array bound to svar2 >> - >> >> Start tasks. >> >> h = start_stasks(shmcfg) >> ntasks = count_stasks(h) >> >> - >> >> The tasks are started and symbols pointing to shared memory segments >> are added as task local storage. A handle is returned. >> - >> >> The shared memory segments are also mapped in the current tasks local >> storage. >> - >> >> NOTE: If nprocs() > 1, then only the Worker Julia processes are used >> to start the Server Tasks, i.e., if nprocs() == 5, then ntasks above would >> be 4. >> - >> >> Prepare arguments for our pmap call >> >> offset_list = [i for i in 1:ntasks] >> ntasks_list = [ntasks for i in 1:ntasks] >> >> - >> >> Execute our function in parallel. >> >> resp = pmap_stasks(h, (offset, ntasks) -> begin >> # get local refernces to shared memory mapped arrays >> svar1 = task_local_storage(:svar1) >> svar2 = task_local_storage(:svar2) >> >> mypid = myid() >> for x in offset:ntasks:64*1024 >> svar1[x] = mypid >> end >> >> true >> >> end, >> >> offset_list, ntasks_list) >> >> - >> >> Access shared memory segments and view changes >> >> svar1 = task_local_storage(:svar1) >> println(svar1) >> >> svar1 will print the values as updated by the Server Tasks. >> >> - Finally stop the tasks >> >> stop_stasks(h, shmcfg) >> >> This causes all tasks to be stopped and the shared memory unmapped. >> >> >>
