My older, non-multithreaded, version works by dividing files into batches
in advance but this means that there may be a 10-20% difference in task
times because we do not know in advance how long it takes to process each
file. The advantage of the multithreaded version is that the variance
between threads is minimal since all threads are pulling from the same
stack.
I should probably re-iterate that the code worked a few weeks ago and has
only started failing lately.
On Sat, Aug 27, 2022 at 8:47 AM bill lam <bbill....@gmail.com> wrote:
> I am not sure why you used mutex. If you collect all file names in a
> subdirectory and divide it into 15 batches (apparently you want 15 tasks)
> and feed each batch to a different tasks then there won't be any
> collisions.
>
>
> On Sat, 27 Aug 2022 at 11:50 AM Devon McCormick <devon...@gmail.com>
> wrote:
>
> > Hi,
> > So, here's some code I put together over the past few months to use
> > multi-threading to parallelize a time-consuming process.
> >
> > We start with the top-level function:
> >
> > NB.* startFlipping: wrapper to set up for "flipN" by creating threads,
> > setting up MUTEX,
> > NB. moving to chip photo dir and flipping them on the chip.
> > startFlipping=: 3 : 0
> > 12 startFlipping y
> > :
> > root=. y,':/DCIM/' NB. y is drive letter of photo chip.
> > subds=. 0{"1 dir root,'*.' NB. Process all subdirs' photos
> > svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0 NB. Fast mutex
> > for_sd. subds do. sd=. ;sd NB. Process each sub-dir
> > 1!:44 ] root,sd
> > if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1''
> > tms=. flipN x<.15 NB. Show time, run x threads, no more
> > than 15.
> > tm=. tm,~6!:1''
> > end.
> > end.
> > 1!:44 svdir
> > tm;tms
> > NB.EG tms=. 12 startFlipping 'F' NB. Run 12 threads for all .JPGs on F:
> > )
> >
> > The thread-creation utility:
> > createThreads=: 3 : '{{ 0 T. '''' }} ^:y]'''''
> >
> > Now we look at the cover for the core routine where we spin off the
> threads
> > in a for loop:
> >
> > flipN=: 3 : 0
> > tms=. i.0
> > while. y<#FLS do. pyx=. i.0
> > for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end.
> > tms=. tms,-/&>_2{.&.>pyx
> > end.
> > NB.EG tms=. flipN 10 NB. Run on 10 threads
> > )
> >
> > This uses a "for" rather than "each" because, in testing, "for" seemed
> less
> > likely to crash.
> >
> > Finally, the deepest function that concerns multi-threaders:
> >
> > NB.* flip1Photo: flip a single photo.
> > flip1Photo=: 3 : 0
> > assert. *./nameExists &> 'MUTEX';'FLS' NB. Need these 2 globals:
> > FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0 NB. Fast mutex
> > tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1'' NB. Current thread
> > number is "tn". "tm" is session start time.
> > if. 0<#FLS do.
> > myFl=. 'ThisFL',tntxt [ 11 T. MUTEX NB. Lock mutex
> > (nms)=: split FLS [ nms=. myFl,' FLS' NB. Take 1 filename from
> > list
> > 13 T. MUTEX NB. Unlock mutex
> > if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end.
> > end.
> > tn,(6!:1''),tm NB. Thread #, end, start session time.
> > NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1 NB. 2 threads
> > )
> >
> > This concentrates all mutex-handling into one place and uses the thread
> > number to try to avoid collisions. If you wanted to try the code above,
> > you could replace "flipPhotoByOrientation" with anything of your choice.
> >
> > Anyway, about a week or so ago, this code seemed to (mostly) work but
> now I
> > get this with only a second or so delay:
> > 6!:2 'tms=. 12 startFlipping ''F''' NB. Run 12 threads for all .JPGs
> on
> > F:
> >
> > Process shell<1> exited abnormally with code 5
> >
> > I'm afraid I may be on the bleeding edge here but does anyone have any
> > ideas? I've done a lot of testing and gotten a few things to work a
> little
> > but not for very long. I used to be able to process hundreds of photos
> > with this code.
> >
> > Any help would be appreciated. The above code should be enough to
> > re-produce the issue once "flipPhotoByOrientation" has been replaced by
> > some target function, preferably one that takes a few seconds to run.
> >
> > Thanks,
> >
> > Devon
> > --
> >
> > Devon McCormick, CFA
> >
> > Quantitative Consultant
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
> >
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
--
Devon McCormick, CFA
Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
