Thank you all for your answers, I am taking off soon to continue a vacation but I have plenty to absorb when I get back.
@Devon Apparently both, Marshall and you, use fork_jtask_, and Marshall's approach seems close to what I had in mind. @Julian Your latest version looks pretty neat. @Jan-Pieter That seems to be a useful full-blown server. @Raul and Skip I remember that you mentioned that project some time ago but this puts it in a new perspective (to me). On Tue, Apr 6, 2021 at 10:54 PM Jose Mario Quintana < [email protected]> wrote: > Excellent! Thank you. > > On Tue, Apr 6, 2021 at 10:10 PM Devon McCormick <[email protected]> > wrote: > >> I have some old references with fairly detailed code here - >> >> https://code.jsoftware.com/wiki/Community/Conference2012/Talks/ParallelSimulationInJ >> - and here - >> >> https://code.jsoftware.com/wiki/User:Devon_McCormick/ParallelizedJCodeExamples >> . The latter one, though older, is more complete. I don't think I have >> changed the code much since I wrote this. The only major change I can >> think of is in the routine that monitors the multiple J processes: I use a >> different underlying process monitor, either "pslist" or "tasklist" (both >> for Windows). Also, I have played around with running multiple copies of >> jconsole with distinct names, like J8Pll00.exe, J8Pll01.exe, and so on. >> This is not necessary but I had the idea I might want to track the >> separate >> processes, so giving them distinct names distinguishes them in a process >> monitor. >> >> The first link has some caveats about peculiarities I uncovered: when I >> shell out multiple processes, they seem to be linked into the same process >> space even though they show up as independent instances of jconsole on a >> process monitor. This seems to be relevant mostly when there are errors >> as >> one has to peel back the stack manually, one process at a time; they are >> distinguishable by having different ARGV_z_ values. >> >> The major change I have wanted to make is to use sockets so that a central >> co-ordinator can parcel out the work in small pieces to a set of listening >> processes. This would better even out the workload for what I'm doing >> (flipping photos upright). The way it works now, the central process >> pre-allocates the work to each process it spins off, so there is always >> one >> that takes longer than all the others because it happened to get more >> work. However, the difference between the fastest and slowest process is >> not so great that I'm highly motivated to do this work, as interesting as >> it would be, because the estimated gain is only on the order of a few >> percent, 10% at the most. >> >> One hack in this code is that I do not know how to dynamically figure out >> how many cores I have available, so I have set up a table with my machine >> names and how many cores each has available. I've found that it pays to >> spin off one less process than there are cores so that the machine is >> still >> usable while the routines are running. By cores, I mean "virtual" cores >> so >> my current 10-core (Intel i9-10900F) machine can run 20 CPU-hungry >> processes, or 19 if I want to be able to do anything else on the machine >> while they are running. >> >> What I've done is coarse-grained parallelism. Note that Marshall Lochbaum >> presented a fine-grained approach at the 2012 J conference that is >> complementary to this but I have not joined his work with my own. >> >> Please feel free to play around with this and ask questions. >> >> >> On Tue, Apr 6, 2021 at 8:36 PM Jose Mario Quintana < >> [email protected]> wrote: >> >> > Do you have handy a link to where can find your routines? It is >> probably >> > close to what I have in mind given your description. >> > >> > On Tue, Apr 6, 2021 at 8:17 PM Devon McCormick <[email protected]> >> wrote: >> > >> > > My home-made parallelization routines spin off multiple copies of J, >> not >> > > using fork explicitly, but it does give significant performance >> > > improvement. It's very simple but works well enough on multi-cores >> that >> > > I've never been motivated enough to try to improve it. >> > > >> > > On Tue, Apr 6, 2021 at 7:47 PM Jose Mario Quintana < >> > > [email protected]> wrote: >> > > >> > > > > $ j -js "exit echo 2 [ (fork&cd bind '') '' [ load >> > > > > 'data/jd/server/fork'" >> > > > > 2 >> > > > > 2 >> > > > >> > > > It seems to me that the above construction works for the UNIX family >> > but >> > > > not for Windows; at least, I managed to run a version of the above >> in a >> > > > very basic BusyBox system but I could not figure out how to run a >> > > > version of it in Windows 10. Am I wrong? (Admittedly, my knowledge >> > > > regarding this matter is very limited.) >> > > > >> > > > > Practical, non-destructive use fork probably has a bunch of >> caveats, >> > > > > but it does also in C programs. >> > > > >> > > > Imagine, for instance, that one wants to evaluate hundreds of times >> an >> > > > expensive arbitrary verb (u), that takes minutes to produce a single >> > > value, >> > > > to plot the verb. In an ideal J world, u("_1) or u(&.>) could be >> used >> > to >> > > > run the evaluations in parallel in minutes as opposed to run them >> > > serially >> > > > in hundreds of minutes. Back to reality, I can find (I think) a >> > > cumbersome >> > > > way, using fork_jtask_, to save significant time when the computer >> has >> > a >> > > > multi-core processor running Windows or a UNIX family OS. However, >> I >> > > > wonder how the experts would attack this kind of problem... >> > > > >> > > > >> > > > >> > > > On Thu, Apr 1, 2021 at 6:12 PM Julian Fondren < >> > [email protected]> >> > > > wrote: >> > > > > >> > > > > A fork bomb is more suited to POSIX fork, which J can use: >> > > > > >> > > > > NB. you might have to reboot if you run this >> > > > > load 'data/jd/server/fork' >> > > > > [ F. (fork&cd bind '') '' >> > > > > >> > > > > Tested separately: >> > > > > >> > > > > $ j -js "exit echo 2 [ (fork&cd bind '') '' [ load >> > > > > 'data/jd/server/fork'" >> > > > > 2 >> > > > > 2 >> > > > > >> > > > > echoing 2 twice, from the two J processes, before they both exit. >> > > > > >> > > > > [ F. (echo bind 2) '' >> > > > > >> > > > > echoing 2 until interrupted. >> > > > > >> > > > > >> > > > > Practical, non-destructive use fork probably has a bunch of >> caveats, >> > > > > but it does also in C programs. >> > > > > >> > > > > On 2021-04-01 16:08, Jose Mario Quintana wrote: >> > > > > > Personally, I prefer the version where although the life of the >> > > > > > individuals >> > > > > > is ephemeral the species survives a lot longer, as it occurs in >> > > nature. >> > > > > > Either way, looking at the structure of the verb fork_jtask_ and >> > its >> > > > > > components, it seems to me that this is a kind of task far more >> > > > > > suitable to >> > > > > > C than J. >> > > > > > >> > > > > > >> > > > > > >> > > > > > On Wed, Mar 31, 2021 at 10:17 PM Raul Miller < >> > [email protected]> >> > > > > > wrote: >> > > > > > >> > > > > >> Sure, and here's a c program which will run into similar >> resource >> > > > > >> limits: >> > > > > >> >> > > > > >> main() { >> > > > > >> while (1) { >> > > > > >> fork(); >> > > > > >> } >> > > > > >> } >> > > > > >> >> > > > > >> This issue was probably one of the motivations for the ulimit >> > > command >> > > > > >> (which people almost never use, nowadays, because we have long >> > since >> > > > > >> learned to expect distributed programs to be well behaved). >> > > > > >> >> > > > > >> Take care, >> > > > > >> >> > > > > >> -- >> > > > > >> Raul >> > > > > >> >> > > > > >> On Wed, Mar 31, 2021 at 6:39 PM Jose Mario Quintana >> > > > > >> <[email protected]> wrote: >> > > > > >> > >> > > > > >> > For some reason, probably the pandemic, recent posts >> regarding >> > the >> > > > verb >> > > > > >> > fork_jtask_ evoked old memories. In the late '70s, while >> > reading >> > > a >> > > > > >> passage >> > > > > >> > in a book describing Von Newman's scheme for constructing >> > > > > >> self-replicating >> > > > > >> > machines, I realized I could design a self-replicating >> process >> > > > capable of >> > > > > >> > running in the computer environment at work. The computer >> was a >> > > > > >> Burroughs >> > > > > >> > B6700 and it had enabled the Inter Process Communication >> (IPC) >> > > > facility >> > > > > >> > which allowed a process to run another process. I wrote a >> tiny >> > > > program >> > > > > >> and >> > > > > >> > showed it as a curiosity to a few of my colleagues telling >> them >> > > that >> > > > it >> > > > > >> > would likely overwhelm the computer; but, for the same >> reason, I >> > > > could >> > > > > >> not >> > > > > >> > test it. >> > > > > >> > >> > > > > >> > Shortly after I went to work for another institution and, in >> the >> > > > early >> > > > > >> > '80s, I moved from Mexico to England and I bought a little >> > > > microcomputer >> > > > > >> > called Sinclair QL. It had a multitasking OS called QDOS >> and a >> > > > BASIC >> > > > > >> > variant called SuperBASIC which was also the QDOS' >> command-line >> > > > > >> > interpreter. So, I rewrote and ran a version of my tiny >> program >> > > > and, as >> > > > > >> > expected, the only way out was to, literally, pull-the-plug. >> > > > > >> > (Incidentally, the machine which looked almost like a >> keyboard >> > was >> > > > also >> > > > > >> > capable to run QL APL, which was a special version of >> MicroAPL's >> > > > > >> APL.68000.) >> > > > > >> > >> > > > > >> > I had swamped not only j but also the OS a few times before, >> but >> > > > never >> > > > > >> > intentionally. So, this is a first for me, the following >> > fleeting >> > > > > >> > script (beware of line-wrapping) runs in an earlier custom >> > version >> > > > of >> > > > > >> the j >> > > > > >> > interpreter on Windows 10 but it should be able to run in the >> > > latest >> > > > and >> > > > > >> > greatest public versions of j and also on other platforms >> > > (changing >> > > > what >> > > > > >> > needs to be changed); however, my strong advice, unless one >> > likes >> > > to >> > > > live >> > > > > >> > dangerously, is: >> > > > > >> > >> > > > > >> > DO NOT RUN IT! >> > > > > >> > >> > > > > >> > NB. Saved as J:/temp/Virus.ijs >> > > > > >> > >> > > > > >> > (2!:55)@:_:@:(([fork_jtask_)^:2) '"J:/Program >> > Files/J/bin/jqt.exe" >> > > > > >> > "J:/temp/Virus.ijs"' >> > > > > >> > >> > > > > >> > >> > > > > >> > PS. Many years later while visiting an old friend in New >> York, >> > > who >> > > > used >> > > > > >> to >> > > > > >> > be a member of the staff operating the B6700, he told me that >> > one >> > > of >> > > > the >> > > > > >> > most stressful times ever at work was when the B6700 suddenly >> > kept >> > > > > >> crashing >> > > > > >> > and crashing for a few days, even missing a payroll deadline. >> > The >> > > > staff >> > > > > >> > and the Burroughs technicians could not find anything wrong >> with >> > > the >> > > > > >> > hardware. The issue was that the system was too clever, >> after a >> > > > crash it >> > > > > >> > would automatically restart all the processes which were >> > > > interrupted. >> > > > > >> > Immediately after identifying the culprit, the sneaky tiny >> > program >> > > > which >> > > > > >> > was very familiar to me, the general access to the IPC >> facility >> > > was >> > > > > >> > disabled... >> > > > > >> > >> > > > > >> > Long live the verb fork_jtask_! :) >> > > > > >> > >> > > > >> ---------------------------------------------------------------------- >> > > > > >> > For information about J forums see >> > > > http://www.jsoftware.com/forums.htm >> > > > > >> >> > > ---------------------------------------------------------------------- >> > > > > >> For information about J forums see >> > > > http://www.jsoftware.com/forums.htm >> > > > > >> >> > > > > > >> > > ---------------------------------------------------------------------- >> > > > > > For information about J forums see >> > > http://www.jsoftware.com/forums.htm >> > > > > >> > ---------------------------------------------------------------------- >> > > > > For information about J forums see >> > http://www.jsoftware.com/forums.htm >> > > > >> ---------------------------------------------------------------------- >> > > > For information about J forums see >> http://www.jsoftware.com/forums.htm >> > > > >> > > >> > > >> > > -- >> > > >> > > Devon McCormick, CFA >> > > >> > > Quantitative Consultant >> > > ---------------------------------------------------------------------- >> > > For information about J forums see >> http://www.jsoftware.com/forums.htm >> > > >> > ---------------------------------------------------------------------- >> > For information about J forums see http://www.jsoftware.com/forums.htm >> > >> >> >> -- >> >> Devon McCormick, CFA >> >> Quantitative Consultant >> ---------------------------------------------------------------------- >> For information about J forums see http://www.jsoftware.com/forums.htm >> > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
