Hi,
So, here's some code I put together over the past few months to use
multi-threading to parallelize a time-consuming process.
We start with the top-level function:
NB.* startFlipping: wrapper to set up for "flipN" by creating threads,
setting up MUTEX,
NB. moving to chip photo dir and flipping them on the chip.
startFlipping=: 3 : 0
12 startFlipping y
:
root=. y,':/DCIM/' NB. y is drive letter of photo chip.
subds=. 0{"1 dir root,'*.' NB. Process all subdirs' photos
svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0 NB. Fast mutex
for_sd. subds do. sd=. ;sd NB. Process each sub-dir
1!:44 ] root,sd
if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1''
tms=. flipN x<.15 NB. Show time, run x threads, no more
than 15.
tm=. tm,~6!:1''
end.
end.
1!:44 svdir
tm;tms
NB.EG tms=. 12 startFlipping 'F' NB. Run 12 threads for all .JPGs on F:
)
The thread-creation utility:
createThreads=: 3 : '{{ 0 T. '''' }} ^:y]'''''
Now we look at the cover for the core routine where we spin off the threads
in a for loop:
flipN=: 3 : 0
tms=. i.0
while. y<#FLS do. pyx=. i.0
for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end.
tms=. tms,-/&>_2{.&.>pyx
end.
NB.EG tms=. flipN 10 NB. Run on 10 threads
)
This uses a "for" rather than "each" because, in testing, "for" seemed less
likely to crash.
Finally, the deepest function that concerns multi-threaders:
NB.* flip1Photo: flip a single photo.
flip1Photo=: 3 : 0
assert. *./nameExists &> 'MUTEX';'FLS' NB. Need these 2 globals:
FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0 NB. Fast mutex
tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1'' NB. Current thread
number is "tn". "tm" is session start time.
if. 0<#FLS do.
myFl=. 'ThisFL',tntxt [ 11 T. MUTEX NB. Lock mutex
(nms)=: split FLS [ nms=. myFl,' FLS' NB. Take 1 filename from
list
13 T. MUTEX NB. Unlock mutex
if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end.
end.
tn,(6!:1''),tm NB. Thread #, end, start session time.
NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1 NB. 2 threads
)
This concentrates all mutex-handling into one place and uses the thread
number to try to avoid collisions. If you wanted to try the code above,
you could replace "flipPhotoByOrientation" with anything of your choice.
Anyway, about a week or so ago, this code seemed to (mostly) work but now I
get this with only a second or so delay:
6!:2 'tms=. 12 startFlipping ''F''' NB. Run 12 threads for all .JPGs on
F:
Process shell<1> exited abnormally with code 5
I'm afraid I may be on the bleeding edge here but does anyone have any
ideas? I've done a lot of testing and gotten a few things to work a little
but not for very long. I used to be able to process hundreds of photos
with this code.
Any help would be appreciated. The above code should be enough to
re-produce the issue once "flipPhotoByOrientation" has been replaced by
some target function, preferably one that takes a few seconds to run.
Thanks,
Devon
--
Devon McCormick, CFA
Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
