What release version did it work on?
Henry Rich
On 8/27/2022 1:46 PM, Devon McCormick wrote:
My older, non-multithreaded, version works by dividing files into batches
in advance but this means that there may be a 10-20% difference in task
times because we do not know in advance how long it takes to process each
file. The advantage of the multithreaded version is that the variance
between threads is minimal since all threads are pulling from the same
stack.
I should probably re-iterate that the code worked a few weeks ago and has
only started failing lately.
On Sat, Aug 27, 2022 at 8:47 AM bill lam <bbill....@gmail.com> wrote:
I am not sure why you used mutex. If you collect all file names in a
subdirectory and divide it into 15 batches (apparently you want 15 tasks)
and feed each batch to a different tasks then there won't be any
collisions.
On Sat, 27 Aug 2022 at 11:50 AM Devon McCormick <devon...@gmail.com>
wrote:
Hi,
So, here's some code I put together over the past few months to use
multi-threading to parallelize a time-consuming process.
We start with the top-level function:
NB.* startFlipping: wrapper to set up for "flipN" by creating threads,
setting up MUTEX,
NB. moving to chip photo dir and flipping them on the chip.
startFlipping=: 3 : 0
12 startFlipping y
:
root=. y,':/DCIM/' NB. y is drive letter of photo chip.
subds=. 0{"1 dir root,'*.' NB. Process all subdirs' photos
svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0 NB. Fast mutex
for_sd. subds do. sd=. ;sd NB. Process each sub-dir
1!:44 ] root,sd
if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1''
tms=. flipN x<.15 NB. Show time, run x threads, no more
than 15.
tm=. tm,~6!:1''
end.
end.
1!:44 svdir
tm;tms
NB.EG tms=. 12 startFlipping 'F' NB. Run 12 threads for all .JPGs on F:
)
The thread-creation utility:
createThreads=: 3 : '{{ 0 T. '''' }} ^:y]'''''
Now we look at the cover for the core routine where we spin off the
threads
in a for loop:
flipN=: 3 : 0
tms=. i.0
while. y<#FLS do. pyx=. i.0
for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end.
tms=. tms,-/&>_2{.&.>pyx
end.
NB.EG tms=. flipN 10 NB. Run on 10 threads
)
This uses a "for" rather than "each" because, in testing, "for" seemed
less
likely to crash.
Finally, the deepest function that concerns multi-threaders:
NB.* flip1Photo: flip a single photo.
flip1Photo=: 3 : 0
assert. *./nameExists &> 'MUTEX';'FLS' NB. Need these 2 globals:
FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0 NB. Fast mutex
tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1'' NB. Current thread
number is "tn". "tm" is session start time.
if. 0<#FLS do.
myFl=. 'ThisFL',tntxt [ 11 T. MUTEX NB. Lock mutex
(nms)=: split FLS [ nms=. myFl,' FLS' NB. Take 1 filename from
list
13 T. MUTEX NB. Unlock mutex
if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end.
end.
tn,(6!:1''),tm NB. Thread #, end, start session time.
NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1 NB. 2 threads
)
This concentrates all mutex-handling into one place and uses the thread
number to try to avoid collisions. If you wanted to try the code above,
you could replace "flipPhotoByOrientation" with anything of your choice.
Anyway, about a week or so ago, this code seemed to (mostly) work but
now I
get this with only a second or so delay:
6!:2 'tms=. 12 startFlipping ''F''' NB. Run 12 threads for all .JPGs
on
F:
Process shell<1> exited abnormally with code 5
I'm afraid I may be on the bleeding edge here but does anyone have any
ideas? I've done a lot of testing and gotten a few things to work a
little
but not for very long. I used to be able to process hundreds of photos
with this code.
Any help would be appreciated. The above code should be enough to
re-produce the issue once "flipPhotoByOrientation" has been replaced by
some target function, preferably one that takes a few seconds to run.
Thanks,
Devon
--
Devon McCormick, CFA
Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
