Are you by any chance using wd to do the actual flipping? As I recall, that
is not thread-safe. (When we make it threadsafe, it will most likely be by
serialising everything, because qt is not threadsafe; so you wouldn't get a
performance boost there, anyway.) If not, what are you using? It might be
the culprit regardless.
I cannot reproduce any crashes using your code. But I do see one issue: you
are globally reassigning MUTEX in flip1Photo. For me, that (correctly) causes
errors to be thrown (when you attempt to unlock the not-already-locked mutex).
But it might be the culprit, especially if you are not on the latest version
(we used to use pthreads mutexes, which have ... interesting behaviour when
misused. Now using custom mutexes, which behave better and are faster).
Does the crashing go away if you remove that assignment?
A note: this code is quite complicated by the manual partitioning of files to
threads. Have you seen u t.'runastask'? It can automate away much of this
work, and go faster too: just say something like flipPhotoByOrientation
t.'runastask'&>photosList. (If you are feeling particularly fancy, you can
also sort the photos by size before starting, which will reduce starvation in
cases of extreme asymmetry.)
-E
On Fri, 26 Aug 2022, Devon McCormick wrote:
Hi,
So, here's some code I put together over the past few months to use
multi-threading to parallelize a time-consuming process.
We start with the top-level function:
NB.* startFlipping: wrapper to set up for "flipN" by creating threads,
setting up MUTEX,
NB. moving to chip photo dir and flipping them on the chip.
startFlipping=: 3 : 0
12 startFlipping y
:
root=. y,':/DCIM/' NB. y is drive letter of photo chip.
subds=. 0{"1 dir root,'*.' NB. Process all subdirs' photos
svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0 NB. Fast mutex
for_sd. subds do. sd=. ;sd NB. Process each sub-dir
1!:44 ] root,sd
if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1''
tms=. flipN x<.15 NB. Show time, run x threads, no more
than 15.
tm=. tm,~6!:1''
end.
end.
1!:44 svdir
tm;tms
NB.EG tms=. 12 startFlipping 'F' NB. Run 12 threads for all .JPGs on F:
)
The thread-creation utility:
createThreads=: 3 : '{{ 0 T. '''' }} ^:y]'''''
Now we look at the cover for the core routine where we spin off the threads
in a for loop:
flipN=: 3 : 0
tms=. i.0
while. y<#FLS do. pyx=. i.0
for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end.
tms=. tms,-/&>_2{.&.>pyx
end.
NB.EG tms=. flipN 10 NB. Run on 10 threads
)
This uses a "for" rather than "each" because, in testing, "for" seemed less
likely to crash.
Finally, the deepest function that concerns multi-threaders:
NB.* flip1Photo: flip a single photo.
flip1Photo=: 3 : 0
assert. *./nameExists &> 'MUTEX';'FLS' NB. Need these 2 globals:
FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0 NB. Fast mutex
tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1'' NB. Current thread
number is "tn". "tm" is session start time.
if. 0<#FLS do.
myFl=. 'ThisFL',tntxt [ 11 T. MUTEX NB. Lock mutex
(nms)=: split FLS [ nms=. myFl,' FLS' NB. Take 1 filename from
list
13 T. MUTEX NB. Unlock mutex
if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end.
end.
tn,(6!:1''),tm NB. Thread #, end, start session time.
NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1 NB. 2 threads
)
This concentrates all mutex-handling into one place and uses the thread
number to try to avoid collisions. If you wanted to try the code above,
you could replace "flipPhotoByOrientation" with anything of your choice.
Anyway, about a week or so ago, this code seemed to (mostly) work but now I
get this with only a second or so delay:
6!:2 'tms=. 12 startFlipping ''F''' NB. Run 12 threads for all .JPGs on
F:
Process shell<1> exited abnormally with code 5
I'm afraid I may be on the bleeding edge here but does anyone have any
ideas? I've done a lot of testing and gotten a few things to work a little
but not for very long. I used to be able to process hundreds of photos
with this code.
Any help would be appreciated. The above code should be enough to
re-produce the issue once "flipPhotoByOrientation" has been replaced by
some target function, preferably one that takes a few seconds to run.
Thanks,
Devon
--
Devon McCormick, CFA
Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
