On Wed, Mar 10, 2010 at 9:30 PM, Jean-Benoist Leger <[email protected]> wrote:
> Hi,
>
> To evaluate a same functions on parallel processes, there are
> parcellfun, which fork an exec function on forks. But I have some
> problems with parcellfun. With some exemple, when I want to do many
> short computations, that this methods spend more time in I/O than
> computation. See below :
>
>> octave:1> A=reshape(1:1024^2,1024,1024);
>> octave:2> B=reshape(1024^2:-1:1,1024,1024);
>> octave:3> cA=num2cell(A);
>> octave:4> cB=num2cell(B);
>> octave:5> testfun=@(x,y) exp(-x)*exp(-y);
>
>> octave:6> tic; a=cellfun(testfun,cA,cB); toc
>> Elapsed time is 13.2478 seconds.
>
>> octave:7> tic; b=parcellfun(5,testfun,cA,cB); toc
>> parcellfun: 1048576/1048576 jobs done
>> Elapsed time is 159.641 seconds.
>
>
> So, we can see that parcellfun is inefficient for doing this task.
>
> My idea, is create meta-jobs which contains many jobs,
> and exec meta-jobs with parcellfun. A meta-job is the exec of many jobs
> by a traditional cellfun.
>
> So I create a function pcellfun which use parcellfun and cellfun which
> is efficient with many very little jobs, see below with the same
> example:
>
>> octave:8> tic; c=pcellfun(5,testfun,cA,cB); toc
>> parcellfun: 500/500 jobs done
>> Elapsed time is 5.3813 seconds.
>
> When parcellfun is efficient (with bigs jobs), pcellfun have the same
> performance than parcellfun, see :
>
>> octave:1> Mats=rand(1299,300);
>> octave:2> cI=num2cell(1:1000);
>> octave:3> testfun=@(k) max(eig(Mats(k:k+299,:)));
>
>> octave:4> tic; a=cellfun(testfun,cI); toc
>> Elapsed time is 162.455 seconds.
>
>> octave:5> tic; b=parcellfun(5,testfun,cI); toc
>> parcellfun: 1000/1000 jobs done
>> Elapsed time is 53.2233 seconds.
>
>> octave:6> tic; c=pcellfun(5,testfun,cI); toc
>> parcellfun: 500/500 jobs done
>> Elapsed time is 52.9729 seconds.
>
> And pcellfun support multiple inputs, multiple ouputs, 'UniformOutput',
> 'ErrorHandler' as cellfun and parcellfun.
>
> In my pcellfun function I copy some code from parcellfun, so I keep the
> copyright.
>
> To test functions I use octave 3.2.4 on a amd64 X4.
>
> Do you think this function is usefull in octave-forge ?
>
I believe it would be better if you just implemented an option for
parcellfun to do this.
For instance,
b = parcellfun (5, testfun, cl, "ChunkSize", 100)
in fact I've had this on my TODO list for some time, but have not yet
got to doing it.
I think you can basically take your code, make it a subfunction of
parcellfun (perhaps remove some duplicate argument parsing), and then
somewhere after argument parsing of parcellfun just do
if (cellsperjob > 1)
[varargout{:}] = chunk_parcellfun (....)
return
endif
where chunk_parcellfun (or name it whatever you like) will prepare the
chunks, call parcellfun recursively, and distribute results.
We will get automatically the same functionality for pararrayfun in
this manner (since pararrayfun is just a wrapper for parcellfun).
What do you think?
regards
--
RNDr. Jaroslav Hajek, PhD
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Octave-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/octave-dev
