If the alternative is to create a new process for each application, that
is a form of multithreading and you wouldn't expect that threading in J
would be much faster.
J threading shines when:
* you want the threads to share data
* you don't know whether threading will help you or not, and you'd like
to see. It's so easy to make a J verb into a task.
Henry Rich
On 12/23/2022 5:23 PM, Devon McCormick wrote:
Yes - it takes about the same time as the older method where I
pre-allocated the files and created a separate process for each
allocation. However, the current run-time is much less variable than the
older one and slightly faster. I run it a few times a week and it's been
fine for months now. The only recent change was, because I got an
i-phone, I had to add 'jpeg' as a known suffix.
So far this month, I've processed 9,235 files totalling 10.92e9 bytes. The
folder sizes range from 200 million to 3.5 billion bytes with the number of
files per folder ranging from 53 to 2815.
I'm always running on new data.
On Fri, Dec 23, 2022 at 3:42 PM Ak O <akin...@gmail.com> wrote:
Hi Devon,
I hope you are well.
About your photo flipping exercise.
Is it now running stably? (at one point you mentioned that it begab
failing)
How are files distributed among folders?
Approximately how many files and folders are you running your test across?
Are you always using the same data?
Ak.
On Tue., Dec. 20, 2022, 10:34 Devon McCormick, <devon...@gmail.com> wrote:
Hi Ak,
That's a lot of questions. I, too, would like to better understand pyxs.
In the meantime, here are some things I've written up on multithreading:
https://code.jsoftware.com/wiki/NYCJUG/2022-05-10#Trying_to_Multithread,
https://code.jsoftware.com/wiki/NYCJUG/2022-07-12#Multi-threaded_Photo_Flipping
,
and
https://code.jsoftware.com/wiki/NYCJUG/2022-08-09#A_New_Multi-threading_Project
.
It looks like I'm using the "Manager Model" in these exercises.
Good luck!
Devon
On Tue, Dec 20, 2022 at 12:13 PM Ak O <akin...@gmail.com> wrote:
I hope you are all well. Thank you for your hard work on this subject.
I am having difficulty understanding how to correctly apply the T. and
t.
(tcapdot/tdot) Primitives based on the vocabulary documentation pages.
The concept of threading is new to me. The documentation pages (T./t.)
use
many terms that I am unclear on.
I appreciate your efforts to explain them. Thank you for your help.
Ak.
Vocabulary_tdot Page
Reading from the Vocabulary documentation page tdot (at indentation).
i. t. ''"0 i. 5
++-+---+-----+-------+
||0|0 1|0 1 2|0 1 2 3|
++-+---+-----+-------+
In the context of threading, what is this statement meant to
characterize?
Is there a sentence of the form ,'execute some operation in some
distinct
thread (having thread attribute(s)..).'?
[x] u t. n y runs ([x] <@:u y) in an idle thread and returns its
result, which is a pyx. A pyx is a special box. It looks like an atomic
box
and can be used like any other box. When the pyx is opened, the opening
thread will block until the value is available. If the creation of the
pyx
ended in error, that error will be signaled when the pyx is opened.
What is:
->the purpose of a 'pyx'?
Where does a 'pyx' fit into the threading concept?
What is:
->thread?
->idle thread?
How is a thread identified?
The n argument to t. controls execution of the task. The format is
[[<]important parameters][,< keyword[;value]]...
The important parameters are: [threadpool#]
What is:
-> a task?
-> important parameters?
-> [threadpool#]
-> threadpool
-> i. t.(0;('worker'))"_1 i.5
threadpool# indicates which threadpool the task should be assigned
to.
How is a threadpool assigned?
How is a threadpool identified?
Threads are assigned to threadpools when they are created.
How are assigned threadpools identified for created threads?
By default, threadpool 0 is used.
What is the equivalent representation for this statement (i.e [x] u t.
n
y
runs ([x] <@:u y))?
One keyword has been defined.
What is a 'keyword' (conceptually and type)?
Is it predefined?
The worker keyword controls whether the task should be executed in
the
originating thread if there is no waiting thread for it to run in.
How does the keyword control the task executing thread?
u t. 'worker'
u t. (<'worker')
u t. ('worker';1)
What differentiates these forms?
Is it always 'worker' a standin?
A few words on cases where one form might be chosen instead of another.
--->
Reference questions
In trying to understand how to build threading structures (threaded
program), three basic models are given:
(https://hpc-tutorials.llnl.gov/posix/designing_threaded_programs
)
Quote:
Several common models for threaded programs exist:
Manager/worker: a single thread, the manager assigns work to
other
threads, the workers.
Typically, the manager handles all input and parcels out work to
the
other
tasks. At least two forms of the manager/worker model are common:
static
worker pool and dynamic worker pool.
Pipeline: a task is broken into a series of suboperations,
each
of
which is handled
in series, but concurrently, by a different thread. An automobile
assembly
line best describes this model.
Peer: similar to the manager/worker model, but after the main
thread creates other
threads, it participates in the work.
Can you please show or diagram the J primitives used to compose the
three
models?
--->
- - - Please give a Primitives translation from C to J under the
Creating
Pthreads heading, Reference Source (
https://randu.org/tutorials/threads/
).
- - -
- - - Please give a Primitives translation from C to J under the
Pthreads
Attributes heading, Reference Source (
https://randu.org/tutorials/threads/
).
- -
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
--
Devon McCormick, CFA
Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
