> > But here is a problem, suppose you have quite large input set (let's > > say size of N), with which you have to evaluate a function in the > > loop (let's say M times). Now you need to reserve N*M amount of > > memory. While if parfor loop you would need no more then M*(number > > of cpu/cores). In other words 'parcellfun' seems to be memory > > hungry. > > Can you give a more specific example? Big inputs are usually generated > from smaller ones; so you can simply extend the parallelized part to work > with the smaller inputs. I see
Well, unfortunately, generation of that input is quite long, mainly because I did not spend much time trying to vectorize the code. But globals indeed could be passed to parcellfun. Apparently I just had a weird typo somewhere. > that unlike cellfun, parcellfun does not allow auto-expanding scalar > cells as arguments. I'll add that feature. But it can be worked around > using anonymous function; in general there is no need to duplicate > inputs. Could you please show me an example with anonymous function, I definitely unaware of this trick. > > > Unless I miss something, I do not see a way to pass a global > > variable to the 'parcellfun' at least it seems to fail at this stage, also > > 'evalin' does not work as well, probably for the same reason. > > > > Surely you can use a global variable in the function being evaluated. > At least it should work; if you found a bug, please submit an example. As I said above, it is a bug in my test code. My apology for the unchecked results. > The whole Octave is memory hungry, so if you're short of memory, Octave > may be problematic in general. Just for the record, for intensive > computations I use Octave on a machine with 8 CPUs and 16GB RAM, and I > don't think I ever exceeded 2GB. My coding machine is much more humble :) I have just 512 MB of RAM. So we have slightly different definition of memory hungry. Before I find how to put globals to parcellfun, I had to copy quite big matrix for every cell, which first of all eats cpu cycles and secondly memory. Now everything seems to be fine. > > By the way is there a way to learn number of CPU/cores from > > inside of octave? > > > Nothing portable I know of. Generally something equivalent to > > cat /proc/cpuinfo | grep "^processor" | wc -l > > will work on Linux. But even if you can determine the number of CPUs, it > doesn't mean that all of them may be used. In the end, I thought it > wasn't worth including such feature in parcellfun. Usually the user knows > well how many cores he has. At our cluster which I yet to learn how to use, they have quite a mix of hardware, so I do not know in advance which machine will execute the code. Matlabs parfor still has some appealing side, it seems to know about local cpu/cores and can also execute a code in a cluster enviroment on remote cpus (for the extra money though). But in worst case scenario it fall back and behave just like a normal 'for'. But it seems like 'parcellfun' could spread on mosix cluster without extra work as well. May be it make sense for 'cellfun' to call 'parcellfun' if some global switch is toggled by user. Of cause once it is argument compatible (i.e. capable of reiterating scalars as cellfun). -- Eugeniy E. Mikhailov ------------------------------------------------------------------------------ Return on Information: Google Enterprise Search pays you back Get the facts. http://p.sf.net/sfu/google-dev2dev _______________________________________________ Octave-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/octave-dev
