I am not sure why you used mutex. If you collect all file names in a
subdirectory and divide it into 15 batches (apparently you want 15 tasks)
and feed each batch to a different tasks then there won't be any collisions.
On Sat, 27 Aug 2022 at 11:50 AM Devon McCormick <devon...@gmail.com> wrote:
> Hi,
> So, here's some code I put together over the past few months to use
> multi-threading to parallelize a time-consuming process.
>
> We start with the top-level function:
>
> NB.* startFlipping: wrapper to set up for "flipN" by creating threads,
> setting up MUTEX,
> NB. moving to chip photo dir and flipping them on the chip.
> startFlipping=: 3 : 0
> 12 startFlipping y
> :
> root=. y,':/DCIM/' NB. y is drive letter of photo chip.
> subds=. 0{"1 dir root,'*.' NB. Process all subdirs' photos
> svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0 NB. Fast mutex
> for_sd. subds do. sd=. ;sd NB. Process each sub-dir
> 1!:44 ] root,sd
> if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1''
> tms=. flipN x<.15 NB. Show time, run x threads, no more
> than 15.
> tm=. tm,~6!:1''
> end.
> end.
> 1!:44 svdir
> tm;tms
> NB.EG tms=. 12 startFlipping 'F' NB. Run 12 threads for all .JPGs on F:
> )
>
> The thread-creation utility:
> createThreads=: 3 : '{{ 0 T. '''' }} ^:y]'''''
>
> Now we look at the cover for the core routine where we spin off the threads
> in a for loop:
>
> flipN=: 3 : 0
> tms=. i.0
> while. y<#FLS do. pyx=. i.0
> for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end.
> tms=. tms,-/&>_2{.&.>pyx
> end.
> NB.EG tms=. flipN 10 NB. Run on 10 threads
> )
>
> This uses a "for" rather than "each" because, in testing, "for" seemed less
> likely to crash.
>
> Finally, the deepest function that concerns multi-threaders:
>
> NB.* flip1Photo: flip a single photo.
> flip1Photo=: 3 : 0
> assert. *./nameExists &> 'MUTEX';'FLS' NB. Need these 2 globals:
> FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0 NB. Fast mutex
> tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1'' NB. Current thread
> number is "tn". "tm" is session start time.
> if. 0<#FLS do.
> myFl=. 'ThisFL',tntxt [ 11 T. MUTEX NB. Lock mutex
> (nms)=: split FLS [ nms=. myFl,' FLS' NB. Take 1 filename from
> list
> 13 T. MUTEX NB. Unlock mutex
> if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end.
> end.
> tn,(6!:1''),tm NB. Thread #, end, start session time.
> NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1 NB. 2 threads
> )
>
> This concentrates all mutex-handling into one place and uses the thread
> number to try to avoid collisions. If you wanted to try the code above,
> you could replace "flipPhotoByOrientation" with anything of your choice.
>
> Anyway, about a week or so ago, this code seemed to (mostly) work but now I
> get this with only a second or so delay:
> 6!:2 'tms=. 12 startFlipping ''F''' NB. Run 12 threads for all .JPGs on
> F:
>
> Process shell<1> exited abnormally with code 5
>
> I'm afraid I may be on the bleeding edge here but does anyone have any
> ideas? I've done a lot of testing and gotten a few things to work a little
> but not for very long. I used to be able to process hundreds of photos
> with this code.
>
> Any help would be appreciated. The above code should be enough to
> re-produce the issue once "flipPhotoByOrientation" has been replaced by
> some target function, preferably one that takes a few seconds to run.
>
> Thanks,
>
> Devon
> --
>
> Devon McCormick, CFA
>
> Quantitative Consultant
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
