hi all,
in case anyone is interested, I committed today into the "general"
package an initial m-file implementation of parcellfun.
parcellfun is supposed to be able to evaluate a given function for
multiple sets of input arguments using multiple processes.
Given N, the function spawns N subprocesses using fork (), and creates
2*N+1 pipes to communicate with them (actually the pipes come first,
but you knew that). Therefore, it should be (in theory) portable to
any Unix system. Most suitable for systems like GNU/Linux, where fork
() is efficient and pipes are a relatively cheap resource. (Dunno
about Windoze, for example).
The function also depends on two assistant compiled functions, fload
and fsave, that can save/load any Octave variable to/from a binary
stream.
a short demo follows (pseudoinversion of 100 400x400 matrices):
n = 400;
m = 100;
disp ("create 100 random 400x400 matrices");
a = rand (n, n, m);
a = mat2cell (a, n, n, ones (1, m));
a = a(:);
disp ("calculate pseudoinverses - uniprocess");
tic;
p = cellfun (@pinv, a, "UniformOutput", false);
toc
clear p
disp ("calculate pseudoinverses - multiprocess");
tic;
p = parcellfun (2, @pinv, a);
toc
on my Core 2 Duo @2.83 GHz machine, using two processes, I get:
create 100 random 400x400 matrices
calculate pseudoinverses - uniprocess
Elapsed time is 55.6268 seconds.
calculate pseudoinverses - multiprocess
parcellfun: 100/100 jobs done
Elapsed time is 30.3612 seconds.
which amounts to some 91% of theoretical peak scalability (and would
be quite nice if it always worked so).
If you have a multithreaded BLAS, remember to switch multithreading
off before testing.
The code uses dynamic scheduling of individual jobs corresponding to
single cells in the input cell arrays.
If the scheduling overhead is not negligible compared to processing a
single input, data should be partitioned
into bigger chunks in advance.
Feedback is much welcome.
enjoy
--
RNDr. Jaroslav Hajek
computing expert
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Octave-dev mailing list
Octave-dev@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/octave-dev
