Hello Alan, > On 24 May 2021, at 01:31, Alan Hylton <agh...@lehigh.edu> wrote: > > Howdy, > > Thanks for the help! > > Right now, I am generating lists of Lie algebras and I wish to calculate the > dimensions of H^2(G, G) and H^3(G, G) for each G. Since these are all > independent and time-consuming, I wanted to split them up. For now, the > return would just be a 2-dimensional list: [ dimension of H^2, dimension of > H^3 ]. > > Because this is not IO intensive, I have elected to use Christopher > Jefferson's suggested solution, ParListByFork.
Glad that you've found a way which works! > > I am still reading through the documentation for SCSCP, and think I will try > to write as many Bash scripts as possible to automate the creation and > farming of tasks. It seems like the way to go is to have a pair of GAP files > - one for the servers that creates all the necessary structures, and one > "main" that is aware of the lists and orderings to make each server operate > on a particular sublist. In principle, two scripts suffice - one gapd.sh to start one GAP server, and another one to start a farm, as https://github.com/alex-konovalov/scscp-demo example shows. If what you are sending as an argument of a remote procedure call is not "heavy" (e.g. a group id or a list of group ids or a range of group ids numbers to check (e.g. [256,1001,1100] to check groups from [256,1001] to [256,1100]), and what you are getting back is a pair of integers, or a list of such pairs, SCSCP should be easily able to handle that efficiently - and also across multiple machines, e.g. on a cluster, or even on a heterogenous pool of SCSCP servers. It also can resubmit RPC calls which were not computed successfully. > Again, I really appreciate it. I am impressed and humbled by how responsive > and helpful the GAP community is. Thank you & best wishes Alexander > > Best, > Alan > > On Tue, May 18, 2021 at 4:23 AM Alexander Konovalov > <alexander.konova...@st-andrews.ac.uk> wrote: > Hi Alan and Sergio, > > In addition to the SCSCP manual, I have > https://github.com/alex-konovalov/scscp-demo > which I was using for the advanced part of the GAP Software Carpentry-style > lesson > (https://carpentries-incubator.github.io/gap-lesson/). > > You may find it useful to set up things, following its README. In particular, > you > can have a script to set up a "farm" of GAP SCSCP servers in one go. From my > experience, for a good load balancing, to run it on N cores, you can start N > servers and one main GAP process from which you will be calling those servers, > and the performance will be better than (N-1) servers and one main. If you > balance the load well, then the main process will be mostly idle, so you can > utilise the N-th core better. But that may depend on the tasks granularity, > and on the data that you pass between nodes. A case study explaining > optimising > this in one particular case can be found here: > > A. Konovalov and S. Linton. 2010. Parallel computations in modular group > algebras. > In Proceedings of the 4th International Workshop on Parallel and Symbolic > Computation (PASCO '10). Association for Computing Machinery, New York, NY, > USA, > 141–149. DOI:https://doi.org/10.1145/1837210.1837231 > > In your case > > >> master_list := [ [...], [...], ..., [...]]; > >> n := Length(master_list); > >> cores:=15; > > > what are the objects that will be arguments of remote procedure calls, and > what > are the results returned back? > > Best wishes > Alexander > > > > > > > On 18 May 2021, at 07:26, Sergio Siccha <ser...@mathb.rwth-aachen.de> wrote: > > > > Hi Alan, > > > > unfortunately `RunTask` in GAP is more or less only a wrapper for > > `CallFuncListWrap` and is indeed blocking. `RunTask` in GAP only is a > > mock-up of the proper `RunTask` function from HPC-GAP. HPC-GAP didn't > > make it out of alpha stage though and if I'm not mistaken nobody is > > working on it ATM. > > > > I'm not sure whether that's the best option as I have never used it, but > > in principle you should be able to do parallel computations by spawning > > several GAP processes via the SCSCP package, see chapter 8 "Parallel > > computing with SCSCP" of the SCSCP manual. So I guess as long as your > > computations don't need a lot of memory that should be fine. > > > > Hope this helps! :-) > > > > Best, > > Sergio > > > > > > On 18.05.21 07:35, Alan Hylton wrote: > >> Howdy, > >> > >> Suppose I have a list of lists, and I wish to run some time-consuming > >> process on each of these sub-list (each sub-list is independent, so I am > >> not worried about race conditions). > >> > >> I think the easiest way to demonstrate my thought process is with code: > >> > >> I have a list of lists and the number of cores I wish to use - > >> master_list := [ [...], [...], ..., [...]]; > >> n := Length(master_list); > >> cores:=15; > >> > >> I have a time consuming function whose arguments are ranges into > >> master_list - > >> time_consuming_func := function(start_index, stop_index) > >> ... > >> end; > >> > >> I portion out master_list, creating a list a tasks - > >> task_list:=[]; > >> start:=1; > >> for i in [1..cores] do > >> flag:=0; > >> if i <= n mod cores then > >> flag:=1; > >> fi; > >> if i > n then > >> break; > >> fi; > >> len:=Int(n/cores)+flag; > >> > >> Add(task_list, RunTask( time_consuming_func , start, start+len-1)); > >> > >> start:=start+len; > >> od; > >> > >> I had several hopes: > >> 1: I could get a list of tasks, and then use something like TaskFinished to > >> see if each are done > >> 2: Store the result of each time_consuming_func in the global master_list > >> > >> But I ran into one problem: RunTask seems to be blocking. Instead of > >> spawning a process and continuing with my loop, it waits until each task > >> finishes. I considered DelayTask instead of RunTask so that I could just > >> use WaitTask, but DelayTask does not seem to exist (similarly for > >> asynchronous tasks). Is there an alternative? To follow the documentation, > >> I'd like to avoid the lower-level CreateThread if I can. > >> > >> Also, number 2 makes some assumptions on how memory works. Is it actually > >> valid to have a thread working on element i of master_list modify the > >> global master_list[i]? > >> > >> I'd greatly appreciate any insight! > >> > >> Thanks, > >> Alan > >> _______________________________________________ > >> Forum mailing list > >> Forum@gap-system.org > >> https://mail.gap-system.org/mailman/listinfo/forum > >> > > > > _______________________________________________ > > Forum mailing list > > Forum@gap-system.org > > https://mail.gap-system.org/mailman/listinfo/forum > _______________________________________________ Forum mailing list Forum@gap-system.org https://mail.gap-system.org/mailman/listinfo/forum
