Re: [GRASS-dev] r.mapcalc 32-bit ceiling
On Sun, Nov 27, 2011 at 4:57 AM, Andy Wickert andrewwick...@gmail.comwrote: There can be a significant performance hit for doing this. Checking whether the result of an addition overflowed is actually more expensive than the addition itself. Checking whether a multiplication overflowed can be even worse (particularly if you don't have a 64-bit integer type available). -- Glynn Clements gl...@gclements.plus.com Hm. If that's a debilitating issue, I agree. And it should actualy not happen or at least a warning should be given. perhaps the overflow check could be triggered by a flag (e.g., if the user thinks there may be a problem) That would be the minimum - I would even opt for an environmental variable GRASS_CHECK_OVERFLOW which provides overflow checking globally for all functions which can produce an overflow. r.mapcalc would be the obvious candidate, but there are others which *can* produce overflow and which should be checking for overflow as well. GRASS_CHECK_OVERFLOW could have three values: - unset or NO (default): do not check for overflow (the default as it is now) - error: check and raise an error if overflow occurs - warning: check and produce a warning if overflow occurs. or (easier) some information on typing and overflow could be placed on the man page. I don't think that is enough - I solved the problem on my side by using double instead of integer, but it took me some time to figure out why my results were wrong. So: Yes, there should definitely be a warning in the manual (highlighted and in Bold so that even I would have seen it...), but the option to check and raise an error on overflow should be provided. And as an environmental variable, it could be used by all modules. Cheers, Rainer Andy -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Stellenbosch University South Africa Tel : +33 - (0)9 53 10 27 44 Cell: +33 - (0)6 85 62 59 98 Fax (F): +33 - (0)9 58 10 27 44 Fax (D):+49 - (0)3 21 21 25 22 44 email: rai...@krugs.de Skype: RMkrug ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: multithreading v.surf.bspline and others with OpenMP
Hi Hamish,
2011/11/28 Hamish hamis...@yahoo.com:
Hi,
On June 2, 2011, Soeren wrote:
Have a look at:
http://grass.osgeo.org/wiki/OpenMP
There are only a few modules using OpenMP (r.gwflow, r3.gwflow and
r.solute.transport). Parts of the gpde and gmath libraries in grass
6.5 and 7 are parallelized using OpenMP. These libraries provide tests
and benchmarks in the libraries test directories.
I am looking to add OpenMP support to v.surf.bspline and am very much
a beginner with it. Profiling with valgrind and kcachegrind (see the
wiki Bugs page) shows that 99% of the module time was spent in gmath lib's
G_math_cholesky_sband_decomposition(). This looks like low hanging fruit.
So I build grass7 with openmp support and add the following patch:
(see OpenMP ./configure patch in the trac'er)
Index: lib/gmath/solvers_direct_cholesky_band.c
===
--- lib/gmath/solvers_direct_cholesky_band.c (revision 49355)
+++ lib/gmath/solvers_direct_cholesky_band.c (working copy)
@@ -24,6 +24,7 @@
for (i = 0; i rows; i++) {
G_percent(i, rows, 2);
+#pragma omp parallel for schedule (static) private(j, k, sum, end) shared(A,
T, i, bandwidth)
for (j = 0; j bandwidth; j++) {
sum = A[i][j];
/* start = 1 */
and on a 6-core CPU the module runs 3.5x faster. but the resulting map
is just full of NaNs. Running with OMP_NUM_THREADS=1 gives me data again.
a) what have I done wrong? why all NaNs?
Try r49406.
You need to separate the computation for j == 0 and j 0.
b) 3.5x speedup is very nice, but any way to improve on that 40% efficiency
loss?
The speedup is better as larger the band matrix is. But limiting
factor of parallel processing speedup is the the first computation for
j == 0. This operation must be done before the rest can be processed
in parallel. The next limiting factor is the time which the OS needs
to create a thread, unless the OpenMP implementation uses a thread
pool ... .
c) for the many raster modules that do
for (row = 0; row nrows; row++) {
for (col = 0; col ncols; col++) {
I'm guessing it is better to multithread the columns for loop (filling
a row's array in parallel), but to keep writing the raster rows serially.
Yes, much better indeed.
But does the cost of creating and destroying 2-16 threads per row end up
costing too much in terms of create/destroy overhead? IIUC schedule
(static) helps to minimize the cost of creating each thread a little? (??)
This is only useful in case the processing of the columns needs much
more time than the thread creation by the OS, unless a thread pool
.
d) I had a look at 'r.univar -e'; it uses heapsort() which AFAIU is a
sorting method which can not be parallelized.
e) I had a look at r.neighbors, which spends 50% of its time in gather().
I got it to run multithreaded on all cores, but the result was many times
slower than if I just had it run serially. why? too much waiting for n?
Multithreading, especially in case of OpenMP reduction, is only
meaningful in case the data is large enough, otherwise the serial
gathering of n and the thread creation takes much longer then the
computation, unless a thread pool . .
int gather(DCELL * values, int offset)
{
int row, col;
int n = 0;
DCELL *c;
*values = 0;
#pragma omp parallel for private(c, col) reduction(+:n)
for (row = 0; row ncb.nsize; row++)
for (col = 0; col ncb.nsize; col++) {
c = ncb.buf[row][offset + col];
if (ncb.mask !ncb.mask[row][col])
continue;
if (Rast_is_d_null_value(c))
Rast_set_d_null_value(values[n], 1);
else
values[n] = *c;
n++;
}
return n ? n : -1;
}
(I moved DCELL *c out of the inner loop as I suspected allocating a
new variable might be expensive. But it still ran at about the same speed
so maybe it is cheap to do that after all, or maybe the compiler did
something smart)
f) we talked a little about this before, but it would be good to settle
on a uniform name for test suite scripts within a module or libraries
directory. I humbly suggest to use test_suite/. Using profiler/
doesn't cover regression testing, test is too generic, and putting
it in the main module dir introduces too much file clutter IMO.
No problem with test_suite for library tests and benchmarks.
g) v.surf.rst spends 58% of its time in gmath lib's G_ludcmp() and 20% of
its time in __iee754_log(). G_ludcmp() also looks like very low hanging
fruit. (great!)
h) is it possible /or desirable to use (aka outsource) pre-optimized
pre-threaded BLAS or LAPACK libraries for our linear algebra needs?
The GRASS ATLAS wrapper is and example for such an approach. ATLAS can be used,
but in case it is not installed, the default GRASS implementation is
used. Feel free to add new
[GRASS-dev] Fedora 16 6.4 release branch 49406 and RC2 failure
Hi, I first tried 6.4.2 RC2, now also checked out SVN 49406, with the same result. With fresh Fedora 16, building from source (with 6.4.1 the proj arguments were never needed), there is trouble loading the GUI: $ ./configure --prefix=/home/rsb/topics/GRASS/G642RC2 --with-wxwidgets --with-proj-libs=/usr/local/lib --with-proj-include=/usr/local/include --with-proj-share=/usr/local/share/proj $ /home/rsb/topics/GRASS/G642RC2/bin/grass64 -wx Cleaning up temporary files ... Starting GRASS ... __ ___ _____ / / __ \/ | / ___/ ___/ / / _/ ___/ / / __/ /_/ / /| | \__ \\_ \ / / __ / / \__ \ / /_/ / _, _/ ___ |___/ /__/ / / /_/ // / ___/ / \/_/ |_/_/ |_/// \/___/// Welcome to GRASS 6.4.2svn (2011) GRASS homepage: http://grass.osgeo.org/ This version running thru: Bash Shell (/bin/bash) Help is available with the command: g.manual -i See the licence terms with: g.version -c If required, restart the GUI with: g.gui wxpython When ready to quit enter:exit GRASS 6.4.2svn (spearfish60):~/topics/GRASS/grass64_release 3D view mode: libproj.so.0: cannot open shared object file: No such file or directory Traceback (most recent call last): File /home/rsb/topics/GRASS/G642RC2/grass-6.4.2svn/etc/wxpython/wxgui.py, line 65, in module from gui_modules import layertree File /home/rsb/topics/GRASS/G642RC2/grass-6.4.2svn/etc/wxpython/gui_modules/layertree.py, line 44, in module import toolbars File /home/rsb/topics/GRASS/G642RC2/grass-6.4.2svn/etc/wxpython/gui_modules/toolbars.py, line 47, in module from nviz import haveNviz File /home/rsb/topics/GRASS/G642RC2/grass-6.4.2svn/etc/wxpython/gui_modules/nviz.py, line 29, in module import nviz_mapdisp File /home/rsb/topics/GRASS/G642RC2/grass-6.4.2svn/etc/wxpython/gui_modules/nviz_mapdisp.py, line 39, in module from workspace import Nviz as NvizDefault File /home/rsb/topics/GRASS/G642RC2/grass-6.4.2svn/etc/wxpython/gui_modules/workspace.py, line 40, in module import wxnviz File /home/rsb/topics/GRASS/G642RC2/grass-6.4.2svn/etc/wxpython/gui_modules/wxnviz.py, line 58, in module errtype = CFUNCTYPE(UNCHECKED(c_int), String, c_int) NameError: name 'UNCHECKED' is not defined GRASS 6.4.2svn (spearfish60):~/topics/GRASS/grass64_release Any ideas? Worth filing a bug? Is this wx, nviz, ctypes, none of them? Roger -- Roger Bivand Department of Economics, NHH Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43 e-mail: roger.biv...@nhh.no ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Fedora 16 6.4 release branch 49406 and RC2 failure
Roger Bivand wrote: Hi, I first tried 6.4.2 RC2, now also checked out SVN 49406, with the same result. With fresh Fedora 16, building from source (with 6.4.1 the proj arguments were never needed), there is trouble loading the GUI: wxNviz has been disabled only 3 days ago in 6.4, latest svn works for me with Fedora 16. This is not a real solution to [snip] File /home/rsb/topics/GRASS/G642RC2/grass-6.4.2svn/etc/wxpython/gui_modules/wxnviz.py, line 58, in module errtype = CFUNCTYPE(UNCHECKED(c_int), String, c_int) NameError: name 'UNCHECKED' is not defined GRASS 6.4.2svn (spearfish60):~/topics/GRASS/grass64_release Any ideas? Worth filing a bug? Is this wx, nviz, ctypes, none of them? but at least grass starts with gui, although without wxnviz. Markus M ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] [GRASS GIS] #1498: patch proposal: new option for v.category to copy category values from one layer to another
#1498: patch proposal: new option for v.category to copy category values from one layer to another -+-- Reporter: mlennert | Owner: grass-dev@… Type: enhancement | Status: new Priority: normal | Milestone: 7.0.0 Component: Vector | Version: unspecified Keywords: v.category |Platform: Unspecified Cpu: Unspecified | -+-- In the attached patch againts trunk I propose an implementation of a feature in v.category requested regularly in the mailing lists: copying category values from one layer to another. I don't think that it is a very invasive addition, but someone should probably check before we apply this to trunk. It could possibly also go into 6.5 as it doesn't break backwards compatibility. I used the word 'transfer' for this option as the way the parser works I needed a word with a different first letter than the others, and 'c'(opy) was already taken by 'c'(hlayer). On thing I did not put in is a way how to work with type=area. I just exit with a message proposing to use type=centroid to copy category values for areas. If someone wants to implement the same with type=area working, please do so. Moritz -- Ticket URL: http://trac.osgeo.org/grass/ticket/1498 GRASS GIS http://grass.osgeo.org ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Proposed patch: v.category option to copy category values from one layer to another [was: Re: [GRASS-user] Re: two questions on setting a new layer for for a vector]
On 28/11/11 11:28, Moritz Lennert wrote: I agree that it would be helpful to have an opion in v.category to copy the values from one layer to another (i.e. something like option=copy layer=2 sourcelayer=1). I just filed an enhancement request with a proposed patch implementing this (http://trac.osgeo.org/grass/ticket/1498). The proposed implementation works as follows: v.category roadsmajor option=transfer layer=1,2 out=roadsmajor_with_2_layers It would be great if anyone could test it and if one of the devs could have a look at the patch and either apply it directly, or tell me that it's ok to do so. Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: [GRASS GIS] #1498: patch proposal: new option for v.category to copy category values from one layer to another
#1498: patch proposal: new option for v.category to copy category values from one layer to another -+-- Reporter: mlennert | Owner: grass-dev@… Type: enhancement | Status: new Priority: normal | Milestone: 7.0.0 Component: Vector | Version: unspecified Keywords: v.category |Platform: Unspecified Cpu: Unspecified | -+-- Comment(by mmetz): Very useful addition, the transfer option! The attached second patch copies not only the first cat encountered but all cats of the first layer to the second layer. Type = centroid is now automatically activated if type = area and not type=area,centroid. Markus M -- Ticket URL: http://trac.osgeo.org/grass/ticket/1498#comment:1 GRASS GIS http://grass.osgeo.org ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Fedora 16 6.4 release branch 49406 and RC2 failure
On Mon, 28 Nov 2011, Roger Bivand wrote: On Mon, 28 Nov 2011, Markus Metz wrote: Roger Bivand wrote: Hi, I first tried 6.4.2 RC2, now also checked out SVN 49406, with the same result. With fresh Fedora 16, building from source (with 6.4.1 the proj arguments were never needed), there is trouble loading the GUI: wxNviz has been disabled only 3 days ago in 6.4, latest svn works for me with Fedora 16. This is not a real solution to Martin wrote on Friday that he had disabled wxNviz in relbr64 at r49360, so I assumed that by checking out r49406, I would avoid the problem I had seen in the RC2 tarball. Am I missing a dependency that is not being trapped by configure? Why does a post-r49360 checkout from: svn checkout https://svn.osgeo.org/grass/grass/branches/releasebranch_6_4 grass64_release still try to build wxnviz? Where should I look to see which files were changed by r49360 on the server, so that I can compare with the ones I checked out? The file on my checkout and currently shown from: http://svn.osgeo.org/grass/grass/branches/releasebranch_6_4/gui/wxpython/gui_modules/nviz.py has lines 31-33: import wxnviz # disable wxNviz for 6.4.2 # TODO: backport wxNviz from devbr6 *after* releasing 6.4.2 haveNviz = False but this doesn't seem OK for my context. The problem is the failure at: errtype = CFUNCTYPE(UNCHECKED(c_int), String, c_int) NameError: name 'UNCHECKED' is not defined so the import wxnwiz fails, doesn't it? Is just setting haveNviz = False enough? Roger Roger [snip] File /home/rsb/topics/GRASS/G642RC2/grass-6.4.2svn/etc/wxpython/gui_modules/wxnviz.py, line 58, in module errtype = CFUNCTYPE(UNCHECKED(c_int), String, c_int) NameError: name 'UNCHECKED' is not defined GRASS 6.4.2svn (spearfish60):~/topics/GRASS/grass64_release Any ideas? Worth filing a bug? Is this wx, nviz, ctypes, none of them? but at least grass starts with gui, although without wxnviz. Markus M -- Roger Bivand Department of Economics, NHH Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43 e-mail: roger.biv...@nhh.no ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: [GRASS GIS] #1498: patch proposal: new option for v.category to copy category values from one layer to another
#1498: patch proposal: new option for v.category to copy category values from one layer to another -+-- Reporter: mlennert | Owner: grass-dev@… Type: enhancement | Status: new Priority: normal | Milestone: 7.0.0 Component: Vector | Version: unspecified Keywords: v.category |Platform: Unspecified Cpu: Unspecified | -+-- Comment(by mlennert): Replying to [comment:1 mmetz]: Very useful addition, the transfer option! The attached second patch copies not only the first cat encountered but all cats of the first layer to the second layer. Type = centroid is now automatically activated if type = area and not type=area,centroid. Thanks, Markus, for looking into this. Can we apply to trunk ? And what about 6.x ? Moritz -- Ticket URL: http://trac.osgeo.org/grass/ticket/1498#comment:2 GRASS GIS http://grass.osgeo.org ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Fedora 16 6.4 release branch 49406 and RC2 failure
Hi, 2011/11/28 Roger Bivand roger.biv...@nhh.no: import wxnviz # disable wxNviz for 6.4.2 # TODO: backport wxNviz from devbr6 *after* releasing 6.4.2 haveNviz = False hopefully fixed in r49409. Martin -- Martin Landa landa.martin gmail.com * http://geo.fsv.cvut.cz/~landa ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: [GRASS GIS] #1498: patch proposal: new option for v.category to copy category values from one layer to another
#1498: patch proposal: new option for v.category to copy category values from one layer to another -+-- Reporter: mlennert | Owner: grass-dev@… Type: enhancement | Status: new Priority: normal | Milestone: 7.0.0 Component: Vector | Version: unspecified Keywords: v.category |Platform: Unspecified Cpu: Unspecified | -+-- Comment(by mmetz): Replying to [comment:2 mlennert]: Replying to [comment:1 mmetz]: Very useful addition, the transfer option! The attached second patch copies not only the first cat encountered but all cats of the first layer to the second layer. Type = centroid is now automatically activated if type = area and not type=area,centroid. Thanks, Markus, for looking into this. Can we apply to trunk ? And what about 6.x ? I have added the new feature of copying categories in r49413, but with further modifications because the category/layer design of GRASS vectors is quite flexible and thus probably a bit complicated: * A warning is printed if categories already exist in the target layer (where categories are copied to). This could lead to all sorts of complications. I was contemplating a fatal error, but then I decided to follow GRASS policy: allow a lot, warn if it may be dangerous. * There is a new hidden feature for option=transfer, if only one layer number is given for the layer option, the module will find the next free layer number and copy categories to that new layer. No danger of confusion with already existing categories in that layer since it's new. This feature is hidden in the sense that it is documented somewhat further down the manual. Since I was on it, I changed the behaviour of option=chlayer a bit: the layer number of categories is changed (as before) if the current category number does not exist for the target layer, otherwise it is only deleted from the source layer (new). This avoids duplicate category entries of a given geometry object in the target layer. All changes are tested and should be backported to 6.x since they maintain backwards compatibility and fix an obscure but potentially disastrous bug. Markus M -- Ticket URL: http://trac.osgeo.org/grass/ticket/1498#comment:3 GRASS GIS http://grass.osgeo.org ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Fedora 16 6.4 release branch 49406 and RC2 failure
On Mon, 28 Nov 2011, Martin Landa wrote: Hi, 2011/11/28 Roger Bivand roger.biv...@nhh.no: import wxnviz # disable wxNviz for 6.4.2 # TODO: backport wxNviz from devbr6 *after* releasing 6.4.2 haveNviz = False hopefully fixed in r49409. Martin Confirmed, thanks! Roger PS. I also needed NumPy installed to avoid a warning, not caught by configure I think. -- Roger Bivand Department of Economics, NHH Norwegian School of Economics, Helleveien 30, N-5045 Bergen, Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43 e-mail: roger.biv...@nhh.no ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] [GRASS GIS] #1499: v.surf.rst in trunk doesn't create write the output map
#1499: v.surf.rst in trunk doesn't create write the output map
+---
Reporter: hamish | Owner: grass-dev@…
Type: defect | Status: new
Priority: major | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: v.surf.rst |Platform: Linux
Cpu: x86-64 |
+---
Hi,
I can't get v.surf.rst in trunk to create any output.
spearfish:
{{{
g.region -d
g.copy vect=archsites,arch_cp
v.build arch_cp
v.surf.rst in=arch_cp zcol=cat elev=test.rst
Processing segments...
[...]
100%
WARNING: Raster map test.rst not found
ERROR: Unable to write raster maps - try to increase resolution
}}}
no matter what I try, same thing.
??,
Hamish
--
Ticket URL: https://trac.osgeo.org/grass/ticket/1499
GRASS GIS http://grass.osgeo.org
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: [GRASS GIS] #1499: v.surf.rst in trunk doesn't write the output map (was: v.surf.rst in trunk doesn't create write the output map)
#1499: v.surf.rst in trunk doesn't write the output map +--- Reporter: hamish | Owner: grass-dev@… Type: defect | Status: new Priority: major | Milestone: 7.0.0 Component: Vector | Version: svn-trunk Keywords: v.surf.rst |Platform: Linux Cpu: x86-64 | +--- -- Ticket URL: https://trac.osgeo.org/grass/ticket/1499#comment:1 GRASS GIS http://grass.osgeo.org ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] [GRASS GIS] #1500: g.region returns wrong convergence angle
#1500: g.region returns wrong convergence angle
--+-
Reporter: annakrat | Owner: grass-dev@…
Type: defect| Status: new
Priority: normal| Milestone:
Component: Default | Version: svn-trunk
Keywords: g.region convergence |Platform: Unspecified
Cpu: Unspecified |
--+-
When I run
{{{
g.region -n
}}}
in G7 in spearfish dataset, I get a nonsense and every time it's different
value:
{{{
(Tue Nov 29 08:04:55 2011)
g.region -n
convergence angle: -65.370921
(Tue Nov 29 08:04:55 2011) Command finished (0 sec)
(Tue Nov 29 08:04:58 2011)
g.region -n
convergence angle: -112.346011
(Tue Nov 29 08:04:58 2011) Command finished (0 sec)
(Tue Nov 29 08:04:59 2011)
g.region -n
convergence angle: -0.922875
(Tue Nov 29 08:04:59 2011) Command finished (0 sec)
}}}
When I switch to different location (nc_spm_08), it seems to be OK.
In grass 6.5 it works (both locations).
--
Ticket URL: http://trac.osgeo.org/grass/ticket/1500
GRASS GIS http://grass.osgeo.org
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Re: multithreading v.surf.bspline and others with OpenMP
Hamish:
I am looking to add OpenMP support to v.surf.bspline
...
Sören wrote:
Try r49406.
You need to separate the computation for j == 0 and j 0.
nice, thanks.
b) 3.5x speedup is very nice, but any way to improve
on that 40% efficiency loss?
The speedup is better as larger the band matrix is. But limiting
factor of parallel processing speedup is the the first computation for
j == 0. This operation must be done before the rest can be processed
in parallel. The next limiting factor is the time which the OS needs
to create a thread, unless the OpenMP implementation uses a thread
pool ... .
I guess that getting a thread pool typically involves waiting until the
next OS upgrade, or longer?
c) for the many raster modules that do
for (row = 0; row nrows; row++) {
for (col = 0; col ncols; col++)
{
I'm guessing it is better to multithread the columns for loop
(filling a row's array in parallel), but to keep writing the raster
rows serially.
Yes, much better indeed.
... but the parallelizing the row loop would be much better if the
whole array was in memory, or mmap()'d?
But does the cost of creating and destroying 2-16 threads per row
end up costing too much in terms of create/destroy overhead?
IIUC schedule (static) helps to minimize the cost of creating
each thread a little? (??)
This is only useful in case the processing of the columns
needs much more time than the thread creation by the OS, unless a
thread pool .
under the current method of parallelizing the inside loop though,
say for a quad-core CPU with a 1200 cols x 800 rows array, we
get 4 threads per row, each handling 300 columns, and for the task
have created and destroyed 4*800= 3200 threads on a system which
will only handle 4 at a time.
much better (but perhaps harder) would be to parallelize as close to
the main process level as we can, and then only deal with the overhead
of creating/destroying e.g. 4 threads not 3200.
On the otherhand, for OpenCL (I'll work on support for that after the
OpenMP stuff has been committed) a modern GPU may well have 500 cores.
in the case of v.surf.bspline I note it runs using 4-16 subregions for
the tests runs I did. if those could each be sent to their own thread
I think we'd be done (for a few years), without the 40% efficiency loss.
If so, is it then possible to call omp_set_num_threads(1); to tell gmath
lib not to try and parallelize it any more? The fn descr says number of
threads used by default in subsequent parallel sections, so maybe so.
Multithreading, especially in case of OpenMP reduction, is only
meaningful in case the data is large enough, otherwise the serial
gathering of n and the thread creation takes much longer then the
computation, unless a thread pool . .
And even moreso for OpenCL, as copying the data into and the result back
out of video card memory is very very slow.
f) we talked a little about this before, but it would
be good to settle on a uniform name for test suite scripts
...
also it would be good to confirm a standard dataset to use. Generating
fake data on the fly is self-boot strapping, but requires passing
fixed seeds to v.random etc. Otherwise N.C.2008 probably gives a
wider range of possibilities than the much smaller spearfish. (mainly
that spearfish doesn't include lidar data)
any thoughts?
g) v.surf.rst spends 58% of its time in gmath lib's G_ludcmp() and 20%
of its time in __iee754_log(). G_ludcmp() also looks like very low
hanging fruit. (great!)
it also looks like a very similar clone of other code in gmath lib, and
I'd expect BLAS/LAPACK/ATLAS too.
I was able to get v.surf.rst to run faster by putting some pragmas into
G_ludcmp(), but again I wonder if it would be more efficient to concentrate
on parallelizing the module's quadtree segments instead of the inner loops
of the linear algebra. And again, maybe a two step approach: do the libs
now (relatively easy), then later do the segments and have that module code
also switch off threading for its library calls with omp_set_num_threads().
h) is it possible /or desirable to use (aka outsource) pre-optimized
pre-threaded BLAS or LAPACK libraries for our linear algebra needs?
The GRASS ATLAS wrapper is and example for such an approach. ATLAS can
be used, but in case it is not installed, the default GRASS
implementation is used.
Oh, I did not know that was there. We can work on adding it to trunk's
./configure next.
Hamish
ps- I didn't add --with-openmp-includes= to ./configure in my patch, just
--with-openmp-libs=. To use the omp_*() fns I guess omp.h is wanted, and
so I should do that after all?
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
