Unfortunately it looks like it was the same release when it worked as when
it didn't:
Engine: j904/j64avx2/windows
Beta-e: commercial/2022-07-16T19:21:14
Library: 9.04.03
Platform: Win 64
My problem may have to do with peculiarities of running on an SD card as
though it were a regular hard-drive, so we can say this problem is not
reproducible until I come up with a cleaner example or it goes away as
mysteriously as it arrived.
I have another multithreading project I will explore to see if the same
problem occurs there. If it does, it should be simple to reproduce.
Until then, thanks for listening.
On Sat, Aug 27, 2022 at 1:49 PM Henry Rich <henryhr...@gmail.com> wrote:
> What release version did it work on?
>
> Henry Rich
>
> On 8/27/2022 1:46 PM, Devon McCormick wrote:
> > My older, non-multithreaded, version works by dividing files into batches
> > in advance but this means that there may be a 10-20% difference in task
> > times because we do not know in advance how long it takes to process each
> > file. The advantage of the multithreaded version is that the variance
> > between threads is minimal since all threads are pulling from the same
> > stack.
> > I should probably re-iterate that the code worked a few weeks ago and has
> > only started failing lately.
> >
> > On Sat, Aug 27, 2022 at 8:47 AM bill lam <bbill....@gmail.com> wrote:
> >
> >> I am not sure why you used mutex. If you collect all file names in a
> >> subdirectory and divide it into 15 batches (apparently you want 15
> tasks)
> >> and feed each batch to a different tasks then there won't be any
> >> collisions.
> >>
> >>
> >> On Sat, 27 Aug 2022 at 11:50 AM Devon McCormick <devon...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>> So, here's some code I put together over the past few months to use
> >>> multi-threading to parallelize a time-consuming process.
> >>>
> >>> We start with the top-level function:
> >>>
> >>> NB.* startFlipping: wrapper to set up for "flipN" by creating threads,
> >>> setting up MUTEX,
> >>> NB. moving to chip photo dir and flipping them on the chip.
> >>> startFlipping=: 3 : 0
> >>> 12 startFlipping y
> >>> :
> >>> root=. y,':/DCIM/' NB. y is drive letter of photo chip.
> >>> subds=. 0{"1 dir root,'*.' NB. Process all subdirs' photos
> >>> svdir=. 1!:43'' [ createThreads 15 [ MUTEX=: 10 T. 0 NB. Fast
> mutex
> >>> for_sd. subds do. sd=. ;sd NB. Process each sub-dir
> >>> 1!:44 ] root,sd
> >>> if. 0~:#FLS=: 0{"1 dir '*.jpg' do. tm=. 6!:1''
> >>> tms=. flipN x<.15 NB. Show time, run x threads, no
> more
> >>> than 15.
> >>> tm=. tm,~6!:1''
> >>> end.
> >>> end.
> >>> 1!:44 svdir
> >>> tm;tms
> >>> NB.EG tms=. 12 startFlipping 'F' NB. Run 12 threads for all .JPGs on
> F:
> >>> )
> >>>
> >>> The thread-creation utility:
> >>> createThreads=: 3 : '{{ 0 T. '''' }} ^:y]'''''
> >>>
> >>> Now we look at the cover for the core routine where we spin off the
> >> threads
> >>> in a for loop:
> >>>
> >>> flipN=: 3 : 0
> >>> tms=. i.0
> >>> while. y<#FLS do. pyx=. i.0
> >>> for_ix. i. y<.#FLS do. pyx=. pyx,flip1Photo t. '']ix end.
> >>> tms=. tms,-/&>_2{.&.>pyx
> >>> end.
> >>> NB.EG tms=. flipN 10 NB. Run on 10 threads
> >>> )
> >>>
> >>> This uses a "for" rather than "each" because, in testing, "for" seemed
> >> less
> >>> likely to crash.
> >>>
> >>> Finally, the deepest function that concerns multi-threaders:
> >>>
> >>> NB.* flip1Photo: flip a single photo.
> >>> flip1Photo=: 3 : 0
> >>> assert. *./nameExists &> 'MUTEX';'FLS' NB. Need these 2 globals:
> >>> FLS=: 0{"1 dir '*.jpg' [ MUTEX=: 10 T. 0 NB. Fast mutex
> >>> tntxt=. ' '-.~":tn=. 3 T. '' [ tm=. 6!:1'' NB. Current thread
> >>> number is "tn". "tm" is session start time.
> >>> if. 0<#FLS do.
> >>> myFl=. 'ThisFL',tntxt [ 11 T. MUTEX NB. Lock mutex
> >>> (nms)=: split FLS [ nms=. myFl,' FLS' NB. Take 1 filename
> from
> >>> list
> >>> 13 T. MUTEX NB. Unlock mutex
> >>> if. 0~:#>".myFl do. *flipPhotoByOrientation* >".myFl end.
> >>> end.
> >>> tn,(6!:1''),tm NB. Thread #, end, start session time.
> >>> NB.EG flip1Photo t. '']0 [ flip1Photo t. '']1 NB. 2 threads
> >>> )
> >>>
> >>> This concentrates all mutex-handling into one place and uses the thread
> >>> number to try to avoid collisions. If you wanted to try the code
> above,
> >>> you could replace "flipPhotoByOrientation" with anything of your
> choice.
> >>>
> >>> Anyway, about a week or so ago, this code seemed to (mostly) work but
> >> now I
> >>> get this with only a second or so delay:
> >>> 6!:2 'tms=. 12 startFlipping ''F''' NB. Run 12 threads for all
> .JPGs
> >> on
> >>> F:
> >>>
> >>> Process shell<1> exited abnormally with code 5
> >>>
> >>> I'm afraid I may be on the bleeding edge here but does anyone have any
> >>> ideas? I've done a lot of testing and gotten a few things to work a
> >> little
> >>> but not for very long. I used to be able to process hundreds of photos
> >>> with this code.
> >>>
> >>> Any help would be appreciated. The above code should be enough to
> >>> re-produce the issue once "flipPhotoByOrientation" has been replaced by
> >>> some target function, preferably one that takes a few seconds to run.
> >>>
> >>> Thanks,
> >>>
> >>> Devon
> >>> --
> >>>
> >>> Devon McCormick, CFA
> >>>
> >>> Quantitative Consultant
> >>> ----------------------------------------------------------------------
> >>> For information about J forums see http://www.jsoftware.com/forums.htm
> >>>
> >> ----------------------------------------------------------------------
> >> For information about J forums see http://www.jsoftware.com/forums.htm
> >>
> >
>
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
--
Devon McCormick, CFA
Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
