Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Markus Neteler]

On Wed, Nov 4, 2015 at 3:24 AM, Glynn Clements  wrote:
> Markus Neteler wrote:
>
>> > At a minimum, I may change the thread pool code (lib/gis/worker.c) so
>> > that the default number of workers (if WORKERS isn't set) is zero,
>> > i.e. multiple threads will only be used if specifically requested by
>> > the user.
>>
>> Maybe better since this question came up a few times.
>
> Done in r66731.

Backported in r67398 for 7.0.3.

Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Markus Neteler]

On Thu, Oct 22, 2015 at 3:14 AM, Glynn Clements
 wrote:
> Dylan Beaudette wrote:
...
> The MASK issue should have been fixed with r65591 (in trunk, not sure
> if it has been backported to releasebranch_7_0 yet).

Yes, in r65764 about 4 months ago.

> Ultimately, if a module uses pthreads itself, it has to ensure that
> any functions which aren't thread-safe aren't called concurrently from
> multiple threads.
>
> Accessing different maps from multiple threads is "intended" to be
> safe. But the issue with the MASK was overlooked, as was the potential
> for issues with GDAL-linked rasters.
>
> I'm starting to think that r.mapcalc's pthread support to may be more
> trouble than it's worth. Unless the expression is particularly
> complex, r.mapcalc tends to be I/O-bound, and pthreads doesn't really
> help with that.

Given all the related updated in GCC etc., perhaps it is possible to
implement reasonable openMP support now?

> At a minimum, I may change the thread pool code (lib/gis/worker.c) so
> that the default number of workers (if WORKERS isn't set) is zero,
> i.e. multiple threads will only be used if specifically requested by
> the user.

Maybe better since this question came up a few times.

>> > But for GDAL-linked (r.external) rasters, there may be re-entrancy
>> > issues with GDAL itself, meaning that even accessing different maps
>> > from multiple threads is unsafe.
>>
>> This is troubling. Is there anyway to empirically determine what is
>> safe? Last week worked on a project where I was processing daily
>> rasters for a 30 year interval, involving 3 climatic variables. Due to
>> the large number of files, I had to work with "external" data sources.
>> For the most part, everything went as expected when the results of
>> (parallel) r.mapcalc expressions were CELL maps. Not so when the
>> results were FCELL or DCELL maps.
>
> The main problem with GDAL is that it's essentially a collection of
> distinct modules in a common framework. Unless they have (and enforce)
> a policy that GDALRasterIO() must be re-entrant (at least for
> different maps), it may be the case that concurrent access is safe for
> some formats but not others.
>
>> > Fixing that is potentially problematic. r.mapcalc doesn't know whether
>> > its inputs are GDAL-linked (this feature is supposed to be transparent
>> > to modules; there isn't a Rast_is_gdal(int fd) or similar function),
>> > and lib/raster itself doesn't have any knowledge of pthreads (lib/gis
>> > provides functions for managing a thread pool; r.mapcalc/r3.mapcalc
>> > are currently the only modules using these functions, and r3.mapcalc
>> > specifically sets WORKERS=0 because lib/raster3d isn't thread-safe).
>>
>> OK. Does setting "WORKERS=0" do anything when GRASS is compiled
>> without pthreads?
>
> No.
>
> If you're experiencing these problems with WORKERS=0 or with GRASS
> compiled without pthread support, then the problem is something else
> entirely.

(apparently no such issue)


Markus


> --
> Glynn Clements 
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Glynn Clements]

Markus Neteler wrote:

> > At a minimum, I may change the thread pool code (lib/gis/worker.c) so
> > that the default number of workers (if WORKERS isn't set) is zero,
> > i.e. multiple threads will only be used if specifically requested by
> > the user.
> 
> Maybe better since this question came up a few times.

Done in r66731.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Glynn Clements]

Dylan Beaudette wrote:

> > If it's a pthread issue, using "export WORKERS=0" from the shell
> > before running t.rast.mapcalc should make it go away.
> 
> I don't think that it has anything to do with pthreads... Does the
> "--without-pthread" configure argument actually work?

--without-pthread is the default. You can check whether GRASS was
built with pthread support by searching for HAVE_PTHREAD_H in
include/grass/config.h. The configure script only checks for
 (and defines that macro) if --with-pthread is given.

> > r.mapcalc uses a mutex for each map to prevent multiple threads
> > accessing a single map concurrently. Accessing different maps from
> > multiple threads should be safe ... for normal GRASS rasters, and
> > provided that no MASK raster is present (otherwise it ends up
> > accessing the MASK raster from multiple threads concurrently, which
> > results in the "Error reading raster data ..." error, due to a race
> > condition between the lseek() and the read()).
> 
> Got it. This is valuable information. Is it documented anywhere other
> than GRASS-user and GRASS-dev mailing lists? How hard would it be to
> issue a warning when someone tries parallel access to the MASK? It
> would seem that some of the new t.* modules could be affected by this
> reality as many of them support parallel processes. Is it the duty of
> the module author to warn of such pitfalls?

The MASK issue should have been fixed with r65591 (in trunk, not sure
if it has been backported to releasebranch_7_0 yet).

Ultimately, if a module uses pthreads itself, it has to ensure that
any functions which aren't thread-safe aren't called concurrently from
multiple threads.

Accessing different maps from multiple threads is "intended" to be
safe. But the issue with the MASK was overlooked, as was the potential
for issues with GDAL-linked rasters.

I'm starting to think that r.mapcalc's pthread support to may be more
trouble than it's worth. Unless the expression is particularly
complex, r.mapcalc tends to be I/O-bound, and pthreads doesn't really
help with that.

At a minimum, I may change the thread pool code (lib/gis/worker.c) so
that the default number of workers (if WORKERS isn't set) is zero,
i.e. multiple threads will only be used if specifically requested by
the user.

> > But for GDAL-linked (r.external) rasters, there may be re-entrancy
> > issues with GDAL itself, meaning that even accessing different maps
> > from multiple threads is unsafe.
> 
> This is troubling. Is there anyway to empirically determine what is
> safe? Last week worked on a project where I was processing daily
> rasters for a 30 year interval, involving 3 climatic variables. Due to
> the large number of files, I had to work with "external" data sources.
> For the most part, everything went as expected when the results of
> (parallel) r.mapcalc expressions were CELL maps. Not so when the
> results were FCELL or DCELL maps.

The main problem with GDAL is that it's essentially a collection of
distinct modules in a common framework. Unless they have (and enforce)
a policy that GDALRasterIO() must be re-entrant (at least for
different maps), it may be the case that concurrent access is safe for
some formats but not others.

> > Fixing that is potentially problematic. r.mapcalc doesn't know whether
> > its inputs are GDAL-linked (this feature is supposed to be transparent
> > to modules; there isn't a Rast_is_gdal(int fd) or similar function),
> > and lib/raster itself doesn't have any knowledge of pthreads (lib/gis
> > provides functions for managing a thread pool; r.mapcalc/r3.mapcalc
> > are currently the only modules using these functions, and r3.mapcalc
> > specifically sets WORKERS=0 because lib/raster3d isn't thread-safe).
> 
> OK. Does setting "WORKERS=0" do anything when GRASS is compiled
> without pthreads?

No.

If you're experiencing these problems with WORKERS=0 or with GRASS
compiled without pthread support, then the problem is something else
entirely.

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Mon, Oct 19, 2015 at 12:45 PM, Glynn Clements
 wrote:
>
> Dylan Beaudette wrote:
>
>> I can now verify that t.rast.mapcalc is creating some raster maps with
>> corrupt (?) data. Corrupt in the sense that subsequent reading of the
>> maps results in the "Error reading raster data for row ..." error.
>> Just in case anyone is interested, I have opened a ticket for more
>> informative errors raised by lib/raster/get_row.c
>> (https://trac.osgeo.org/grass/ticket/2762).
>>
>> As previously reported, errors seem to occur about 50-60% of the time
>> and _do not_ appear to be related to the number of concurrent
>> t.rast.mapcalc instances.
>
> t.rast.mapcalc relies upon r.mapcalc.

OK, thanks for the clarification. I'll be more specific in future
tests / posts / tickets.

> If it's a pthread issue, using "export WORKERS=0" from the shell
> before running t.rast.mapcalc should make it go away.

I don't think that it has anything to do with pthreads... Does the
"--without-pthread" configure argument actually work?

> r.mapcalc uses a mutex for each map to prevent multiple threads
> accessing a single map concurrently. Accessing different maps from
> multiple threads should be safe ... for normal GRASS rasters, and
> provided that no MASK raster is present (otherwise it ends up
> accessing the MASK raster from multiple threads concurrently, which
> results in the "Error reading raster data ..." error, due to a race
> condition between the lseek() and the read()).

Got it. This is valuable information. Is it documented anywhere other
than GRASS-user and GRASS-dev mailing lists? How hard would it be to
issue a warning when someone tries parallel access to the MASK? It
would seem that some of the new t.* modules could be affected by this
reality as many of them support parallel processes. Is it the duty of
the module author to warn of such pitfalls?

Again, not sure how to implement but happy to test / document.


> But for GDAL-linked (r.external) rasters, there may be re-entrancy
> issues with GDAL itself, meaning that even accessing different maps
> from multiple threads is unsafe.

This is troubling. Is there anyway to empirically determine what is
safe? Last week worked on a project where I was processing daily
rasters for a 30 year interval, involving 3 climatic variables. Due to
the large number of files, I had to work with "external" data sources.
For the most part, everything went as expected when the results of
(parallel) r.mapcalc expressions were CELL maps. Not so when the
results were FCELL or DCELL maps.


> Fixing that is potentially problematic. r.mapcalc doesn't know whether
> its inputs are GDAL-linked (this feature is supposed to be transparent
> to modules; there isn't a Rast_is_gdal(int fd) or similar function),
> and lib/raster itself doesn't have any knowledge of pthreads (lib/gis
> provides functions for managing a thread pool; r.mapcalc/r3.mapcalc
> are currently the only modules using these functions, and r3.mapcalc
> specifically sets WORKERS=0 because lib/raster3d isn't thread-safe).

OK. Does setting "WORKERS=0" do anything when GRASS is compiled
without pthreads?

Thank you,
Dylan


> --
> Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Glynn Clements]

Dylan Beaudette wrote:

> I can now verify that t.rast.mapcalc is creating some raster maps with
> corrupt (?) data. Corrupt in the sense that subsequent reading of the
> maps results in the "Error reading raster data for row ..." error.
> Just in case anyone is interested, I have opened a ticket for more
> informative errors raised by lib/raster/get_row.c
> (https://trac.osgeo.org/grass/ticket/2762).
> 
> As previously reported, errors seem to occur about 50-60% of the time
> and _do not_ appear to be related to the number of concurrent
> t.rast.mapcalc instances.

t.rast.mapcalc relies upon r.mapcalc.

If it's a pthread issue, using "export WORKERS=0" from the shell
before running t.rast.mapcalc should make it go away.

r.mapcalc uses a mutex for each map to prevent multiple threads
accessing a single map concurrently. Accessing different maps from
multiple threads should be safe ... for normal GRASS rasters, and
provided that no MASK raster is present (otherwise it ends up
accessing the MASK raster from multiple threads concurrently, which
results in the "Error reading raster data ..." error, due to a race
condition between the lseek() and the read()).

But for GDAL-linked (r.external) rasters, there may be re-entrancy
issues with GDAL itself, meaning that even accessing different maps
from multiple threads is unsafe.

Fixing that is potentially problematic. r.mapcalc doesn't know whether
its inputs are GDAL-linked (this feature is supposed to be transparent
to modules; there isn't a Rast_is_gdal(int fd) or similar function),
and lib/raster itself doesn't have any knowledge of pthreads (lib/gis
provides functions for managing a thread pool; r.mapcalc/r3.mapcalc
are currently the only modules using these functions, and r3.mapcalc
specifically sets WORKERS=0 because lib/raster3d isn't thread-safe).

-- 
Glynn Clements 
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Wed, Oct 14, 2015 at 9:45 PM, Dylan Beaudette
 wrote:
> On Wed, Oct 14, 2015 at 12:55 PM, Dylan Beaudette
>  wrote:
>> On Wed, Oct 14, 2015 at 10:50 AM, Dylan Beaudette
>>  wrote:
>>> Some additional clues:
>>>
>>> The original stack was 365 maps with 3105 x 7025 cells.
>>>
>>> 1. zooming into a smaller region (30 x 40 cells) and running
>>> t.rast.series 100x resulted in 100 "correct" maps, no errors.
>>>
>>> 2. returning to the full extent and running t.rast.series 30x on the
>>> first 31 maps resulted in 30 "correct" maps, no errors.
>>>
>>> 3. returning to the full extent and running t.rast.series 30x on the
>>> last 31 maps resulted in 30 "correct" maps, no errors
>>>
>>>
>>> So, it seems that t.rast.series (r.series) is throwing an error, or
>>> generating wront output, when when:
>>>
>>> a large set of maps are supplied as input, and, a region that has a
>>> moderate number of total cells.
>>>
>>> Yeah, I know, that isn't very specific. I will try re-compiling with
>>> debugging and no optimization next.
>>>
>>> Dylan
>>>
>>>
>>
>> More data,
>>
>> 1. re-compiled with CFLAGS="-g -Wall":
>>  * Multiple runs of t.rast.series with the full stack (365 maps with
>> 3105 x 7025 cells), no errors.
>>  * each run required about 8.5 minutes to complete
>>
>> 2. re-compiled with  CFLAGS="-O2 -mtune=native -march=native" LDFLAGS="-s":
>>  * 10x tests with full stack, no errors
>>  * each run required about 3.5 minutes
>>
>> 3. re-run original script (see listing below)
>>  * random errors from t.rast.series
>>
>> This doesn't make much sense to me. The only difference between my
>> latest "tests" and the original code is that the input to
>> t.rast.series was static over the course of my "tests", vs. dynamic
>> within the original code (see below). I purposely selected a stack
>> that caused t.rast.series to throw an error for my tests.
>>
>
> OK, this does make sense--t.rast.series (r.series) was not the source
> of the problems. I was able to verify this by running t.univar on the
> output from the previous step:
>
>>   # NOTE: 4 CPUs so that external disk isn't thrashed
>>   gdd_max_C=30
>>   gdd_min_C=10
>>   gdd_base_C=10
>>   t.rast.mapcalc --q --o nprocs=4 input=tmin_subset,tmax_subset
>> output=gdd basename=gdd expr="max(((min(tmax_subset, $gdd_max_C) +
>> max(tmin_subset, $gdd_min_C)) / 2.0) - $gdd_base_C, 0)"
>
> ... which means that t.rast.mapcalc was generating one (or more)
> outputs with some kind of problem, which was then causing t.univar and
> t.rast.series to fail.

I can now verify that t.rast.mapcalc is creating some raster maps with
corrupt (?) data. Corrupt in the sense that subsequent reading of the
maps results in the "Error reading raster data for row ..." error.
Just in case anyone is interested, I have opened a ticket for more
informative errors raised by lib/raster/get_row.c
(https://trac.osgeo.org/grass/ticket/2762).

As previously reported, errors seem to occur about 50-60% of the time
and _do not_ appear to be related to the number of concurrent
t.rast.mapcalc instances.

After some more testing, I have found that t.rast.mapcalc does not
(randomly) generate corrupt maps when the output from the mapcalc
expression results in a CELL type map. Expressions that result in both
FCELL and DCELL seem to trigger the corruption.

Fortunately my current project isn't too discriminating and is fine
with CELL output from t.rast.mapcalc.

I now suspect that this is an overflow issue in t.rast.mapcalc (well
the library functions that it calls) that may or may not be influenced
by the use of files linked via r.external.


> The inputs to t.rast.mapcalc are files that have been registered with
> r.external. I suspect that the multiple concurrent r.mapcalc instances
> may be to blame. I don't have an explanation other than some evidence
> from the last time I encountered this type of issue. The workflow then
> was :
>
> 1. spawn 8 concurrent processes via backgrounding: r.sun -> r.mapcalc
>
> 2. when finished with daily solar models, sum maps with r.series
>
> I would occasionally encounter the "Error reading raster data for row
> xxx" error from r.series in this case and assume that r.series had
> somehow broken the map in question.
>
> It would seem that concurrent use of r.mapcalc may be worth
> investigating... however, it is strange that it only occurs sometimes.

I stand corrected. My previous encounters with the "Error reading
raster data for row ..." error were likely associated with this
related problem, which is now fixed:

http://lists.osgeo.org/pipermail/grass-dev/2015-July/075627.html



> Oddly enough, I didn't have problems with maps generated with the
> following (similar) code:
>
> # spring frost
> # if tmin never drops below 0 before the start of summer, then the
> last "spring frost" is on day 0
> # NOTE: 2 CPUs so that disk isn't thrashed
> t.rast.mapcalc --o -n nprocs=2 input=tmin 

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

Some additional clues:

The original stack was 365 maps with 3105 x 7025 cells.

1. zooming into a smaller region (30 x 40 cells) and running
t.rast.series 100x resulted in 100 "correct" maps, no errors.

2. returning to the full extent and running t.rast.series 30x on the
first 31 maps resulted in 30 "correct" maps, no errors.

3. returning to the full extent and running t.rast.series 30x on the
last 31 maps resulted in 30 "correct" maps, no errors


So, it seems that t.rast.series (r.series) is throwing an error, or
generating wront output, when when:

a large set of maps are supplied as input, and, a region that has a
moderate number of total cells.

Yeah, I know, that isn't very specific. I will try re-compiling with
debugging and no optimization next.

Dylan



On Wed, Oct 14, 2015 at 9:55 AM, Dylan Beaudette
 wrote:
> On Tue, Oct 13, 2015 at 11:58 PM, Moritz Lennert
>  wrote:
>> On 13/10/15 20:43, Dylan Beaudette wrote:
>>>
>>> Dangit... This is strange, just did an 'svn up', make distclean, make,
>>> make install, and now this:
>>>
>>> Welcome to GRASS GIS 7.1.svn (r66487)
>>> GRASS GIS homepage:  http://grass.osgeo.org
>>> This version running through:Bash Shell (/bin/bash)
>>> Help is available with the command:  g.manual -i
>>> See the licence terms with:  g.version -c
>>> Start the GUI with:  g.gui wxpython
>>> When ready to quit enter:exit
>>>
>>> GRASS 7.1.svn (prism):~/src/grass_trunk > g.version -r
>>> GRASS 7.1.svn (2015)
>>> libgis Revision: 64732
>>> libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)
>>>
>>> Has it been so long since I have compiled GRASS that I have missed
>>> something? I see that the libgis is still "old".
>>
>>
>> I've sometimes had to completely erase my source tree and do a fresh svn
>> checkout to get a clean new compile. Not sure why, though.
>>
>> Another thing: you might want to do the make distclean before the svn up.
>>
>> Moritz
>
> Thanks for the tips Moritz,
>
> I have tried your suggestion and still get the same errors.
>
> One more clue in regards to the original "Error reading raster data
> for row xxx" issue:
>
> * about half of the time the results from t.rast.series are correct
> * the other half of the time I get "Error reading raster data for row xxx"
> * about 1 time in 30 runs the resulting map will be created, but will
> have values at the extreme edges of FCELL precision... suggesting some
> kind of overflow.
>
> Dylan
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Tue, Oct 13, 2015 at 11:58 PM, Moritz Lennert
 wrote:
> On 13/10/15 20:43, Dylan Beaudette wrote:
>>
>> Dangit... This is strange, just did an 'svn up', make distclean, make,
>> make install, and now this:
>>
>> Welcome to GRASS GIS 7.1.svn (r66487)
>> GRASS GIS homepage:  http://grass.osgeo.org
>> This version running through:Bash Shell (/bin/bash)
>> Help is available with the command:  g.manual -i
>> See the licence terms with:  g.version -c
>> Start the GUI with:  g.gui wxpython
>> When ready to quit enter:exit
>>
>> GRASS 7.1.svn (prism):~/src/grass_trunk > g.version -r
>> GRASS 7.1.svn (2015)
>> libgis Revision: 64732
>> libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)
>>
>> Has it been so long since I have compiled GRASS that I have missed
>> something? I see that the libgis is still "old".
>
>
> I've sometimes had to completely erase my source tree and do a fresh svn
> checkout to get a clean new compile. Not sure why, though.
>
> Another thing: you might want to do the make distclean before the svn up.
>
> Moritz

Thanks for the tips Moritz,

I have tried your suggestion and still get the same errors.

One more clue in regards to the original "Error reading raster data
for row xxx" issue:

* about half of the time the results from t.rast.series are correct
* the other half of the time I get "Error reading raster data for row xxx"
* about 1 time in 30 runs the resulting map will be created, but will
have values at the extreme edges of FCELL precision... suggesting some
kind of overflow.

Dylan
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Moritz Lennert]

On 13/10/15 20:43, Dylan Beaudette wrote:

Dangit... This is strange, just did an 'svn up', make distclean, make,
make install, and now this:

Welcome to GRASS GIS 7.1.svn (r66487)
GRASS GIS homepage:  http://grass.osgeo.org
This version running through:Bash Shell (/bin/bash)
Help is available with the command:  g.manual -i
See the licence terms with:  g.version -c
Start the GUI with:  g.gui wxpython
When ready to quit enter:exit

GRASS 7.1.svn (prism):~/src/grass_trunk > g.version -r
GRASS 7.1.svn (2015)
libgis Revision: 64732
libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)

Has it been so long since I have compiled GRASS that I have missed
something? I see that the libgis is still "old".


I've sometimes had to completely erase my source tree and do a fresh svn 
checkout to get a clean new compile. Not sure why, though.


Another thing: you might want to do the make distclean before the svn up.

Moritz

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Wed, Oct 14, 2015 at 10:50 AM, Dylan Beaudette
 wrote:
> Some additional clues:
>
> The original stack was 365 maps with 3105 x 7025 cells.
>
> 1. zooming into a smaller region (30 x 40 cells) and running
> t.rast.series 100x resulted in 100 "correct" maps, no errors.
>
> 2. returning to the full extent and running t.rast.series 30x on the
> first 31 maps resulted in 30 "correct" maps, no errors.
>
> 3. returning to the full extent and running t.rast.series 30x on the
> last 31 maps resulted in 30 "correct" maps, no errors
>
>
> So, it seems that t.rast.series (r.series) is throwing an error, or
> generating wront output, when when:
>
> a large set of maps are supplied as input, and, a region that has a
> moderate number of total cells.
>
> Yeah, I know, that isn't very specific. I will try re-compiling with
> debugging and no optimization next.
>
> Dylan
>
>

More data,

1. re-compiled with CFLAGS="-g -Wall":
 * Multiple runs of t.rast.series with the full stack (365 maps with
3105 x 7025 cells), no errors.
 * each run required about 8.5 minutes to complete

2. re-compiled with  CFLAGS="-O2 -mtune=native -march=native" LDFLAGS="-s":
 * 10x tests with full stack, no errors
 * each run required about 3.5 minutes

3. re-run original script (see listing below)
 * random errors from t.rast.series

This doesn't make much sense to me. The only difference between my
latest "tests" and the original code is that the input to
t.rast.series was static over the course of my "tests", vs. dynamic
within the original code (see below). I purposely selected a stack
that caused t.rast.series to throw an error for my tests.

Arg!

Dylan


For the record, here is the script that I have been using:

years=`seq 1981 2010`
for year in $years
  do
  echo $year

  # make a where clause for finding the current year
  wc="strftime('%Y', start_time) = '"$year"'"

  # extract current dataset: "copies" by reference
  t.rast.extract --q --o input=tmin output=tmin_subset
basename=tmin_subset where="$wc"
  t.rast.extract --q --o input=tmax output=tmax_subset
basename=tmax_subset where="$wc"

  # compute GDD on each day (about 36 minutes)
  # values less than 0 are set to 0
  # NOTE: 4 CPUs so that external disk isn't thrashed
  gdd_max_C=30
  gdd_min_C=10
  gdd_base_C=10
  t.rast.mapcalc --q --o nprocs=4 input=tmin_subset,tmax_subset
output=gdd basename=gdd expr="max(((min(tmax_subset, $gdd_max_C) +
max(tmin_subset, $gdd_min_C)) / 2.0) - $gdd_base_C, 0)"

  # save Sonora time-series for later
  # -k flag ensures that the output is in same order as input
  t.rast.list -s gdd columns="id" | parallel -k --gnu r.what map="{}"
points=sonora > qa_qc/sonora-daily-gdd-$year.dat

  # sum GDD for this year: fast from SSD, but only single CPU thread
(3.5 minutes)
  # BUG: crashes randomly
  t.rast.series --q --o in=gdd out=gdd_$year method=sum

  done
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Wed, Oct 14, 2015 at 12:55 PM, Dylan Beaudette
 wrote:
> On Wed, Oct 14, 2015 at 10:50 AM, Dylan Beaudette
>  wrote:
>> Some additional clues:
>>
>> The original stack was 365 maps with 3105 x 7025 cells.
>>
>> 1. zooming into a smaller region (30 x 40 cells) and running
>> t.rast.series 100x resulted in 100 "correct" maps, no errors.
>>
>> 2. returning to the full extent and running t.rast.series 30x on the
>> first 31 maps resulted in 30 "correct" maps, no errors.
>>
>> 3. returning to the full extent and running t.rast.series 30x on the
>> last 31 maps resulted in 30 "correct" maps, no errors
>>
>>
>> So, it seems that t.rast.series (r.series) is throwing an error, or
>> generating wront output, when when:
>>
>> a large set of maps are supplied as input, and, a region that has a
>> moderate number of total cells.
>>
>> Yeah, I know, that isn't very specific. I will try re-compiling with
>> debugging and no optimization next.
>>
>> Dylan
>>
>>
>
> More data,
>
> 1. re-compiled with CFLAGS="-g -Wall":
>  * Multiple runs of t.rast.series with the full stack (365 maps with
> 3105 x 7025 cells), no errors.
>  * each run required about 8.5 minutes to complete
>
> 2. re-compiled with  CFLAGS="-O2 -mtune=native -march=native" LDFLAGS="-s":
>  * 10x tests with full stack, no errors
>  * each run required about 3.5 minutes
>
> 3. re-run original script (see listing below)
>  * random errors from t.rast.series
>
> This doesn't make much sense to me. The only difference between my
> latest "tests" and the original code is that the input to
> t.rast.series was static over the course of my "tests", vs. dynamic
> within the original code (see below). I purposely selected a stack
> that caused t.rast.series to throw an error for my tests.
>

OK, this does make sense--t.rast.series (r.series) was not the source
of the problems. I was able to verify this by running t.univar on the
output from the previous step:

>   # NOTE: 4 CPUs so that external disk isn't thrashed
>   gdd_max_C=30
>   gdd_min_C=10
>   gdd_base_C=10
>   t.rast.mapcalc --q --o nprocs=4 input=tmin_subset,tmax_subset
> output=gdd basename=gdd expr="max(((min(tmax_subset, $gdd_max_C) +
> max(tmin_subset, $gdd_min_C)) / 2.0) - $gdd_base_C, 0)"

... which means that t.rast.mapcalc was generating one (or more)
outputs with some kind of problem, which was then causing t.univar and
t.rast.series to fail.

The inputs to t.rast.mapcalc are files that have been registered with
r.external. I suspect that the multiple concurrent r.mapcalc instances
may be to blame. I don't have an explanation other than some evidence
from the last time I encountered this type of issue. The workflow then
was :

1. spawn 8 concurrent processes via backgrounding: r.sun -> r.mapcalc

2. when finished with daily solar models, sum maps with r.series

I would occasionally encounter the "Error reading raster data for row
xxx" error from r.series in this case and assume that r.series had
somehow broken the map in question.

It would seem that concurrent use of r.mapcalc may be worth
investigating... however, it is strange that it only occurs sometimes.

Oddly enough, I didn't have problems with maps generated with the
following (similar) code:

# spring frost
# if tmin never drops below 0 before the start of summer, then the
last "spring frost" is on day 0
# NOTE: 2 CPUs so that disk isn't thrashed
t.rast.mapcalc --o -n nprocs=2 input=tmin output=spring_frost
basename=spring_frost \
expr="if(start_doy() < 182, if(tmin < 0, start_doy(), 0), null())"

# fall frost
# NOTE: 2 CPUs so that disk isn't thrashed
t.rast.mapcalc --o -n nprocs=2 input=tmin output=fall_frost
basename=fall_frost \
expr="if(start_doy() > 213, if(tmin < 0, start_doy(), 365), null())"

Dylan
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Tue, Oct 13, 2015 at 4:25 PM, Dylan Beaudette
 wrote:
> On Tue, Oct 13, 2015 at 11:06 AM, Markus Neteler  wrote:
>>
>> On Oct 13, 2015 7:01 PM, "Dylan Beaudette" 
>> wrote:
>>>
>>> Hi Markus,
>>>
>>> GRASS version information:
>>>
>>>  ./configure  --without-odbc --without-mysql --with-readline
>>> --with-cxx --enable-largefile --with-freetype
>>> --with-freetype-includes=/usr/include/freetype2 --with-sqlite
>>> --with-python --with-pthread
>>
>> I would not use pthread.
>>
>
> Hi Markus,
>
> I have done some tests after re-compiling _without_ pthreads. So far,
> no errors from r.series. Also I seem to be getting better performance
> form r.mapcalc and t.rast.series, especially when working with files
> loaded via r.external. I don't know enough about pthreads to speculate
> further. Any ideas?
>
> I'll report back when the script finishes, probably 20 hours or so.
>

Reporting back, same error occurring in a seemingly random way.

Dylan
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Moritz Lennert]

On 12/10/15 23:35, Dylan Beaudette wrote:

Hi,

Over the last couple of years I have noticed a very strange raster
corruption (?) issues when using r.series, and now more recently,
t.rast.series. Typically, I'll generate a large number of maps with
r.sun or t.rast.mapcalc and then aggregate the series with r.series or
t.rast.series. About 50% of the time the command runs as expected, the
other half of the time r.series or t.rast.series gives me an error
like this:

Error reading raster data for row xxx (testmap)

After this error, I can no longer perform any kind of operation on map
"testmap" without the dreaded Error reading raster data for row xxx...

The situation was worse when using a MASK map, possibly related to a
similar (fixed?) issue discussed in this thread:

http://lists.osgeo.org/pipermail/grass-dev/2015-July/075627.html

Within that thread, Glynn mentioned that this type of error was
probably related to pthreads and concurrent processes. The temporary
fix entailed:

export WORKERS=0

I have tried this on my machine but the results are the same,
non-deterministic corruption (?) of one input to r.series or
t.rast.series.

I have encountered this error on several disks, mirrored HDDs, single
HDD, and now on an SSD. I don't think that this is a disk problem,
rather, something that r.series or t.rast.series is "doing" to the
files it operates on.

Is there some possibility that one of these commands is leaving a file
"open" or in some kind of intermediate state that prevents subsequent
commands from accessing the file?

I'll try to create a sample dataset to send over. In the meantime is
there any kind of diagnostic information that I can report back with?


Are you using a mask, as was the case in the thread you cite ?

Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

One more piece of data:

This error only occurs when using "method=sum". I have not encountered
this error when using any of the other methods available to r.series
or t.rast.series.

Dylan

On Tue, Oct 13, 2015 at 10:00 AM, Dylan Beaudette
 wrote:
> Hi Markus,
>
> GRASS version information:
>
>  ./configure  --without-odbc --without-mysql --with-readline
> --with-cxx --enable-largefile --with-freetype
> --with-freetype-includes=/usr/include/freetype2 --with-sqlite
> --with-python --with-pthread --with-geos=/usr/local/bin/geos-config
> --without-opencl --with-opencl-includes=/usr/include/CL/
> --with-postgres --with-postgres-includes=/usr/include/postgresql/
> --with-postgres-libs=/usr/lib/
> --with-proj-share=/usr/local/share/proj/
>
> libgis Revision: 64732
> libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)
>
> PROJ.4: 4.8.0
> GDAL/OGR: 2.0.0
> GEOS: 3.4.2
> SQLite: 3.7.9
>
> I find it strange that I have encountered this mysterious error
> (occasionally) over the last couple of years while tracking
> grass_trunk.
>
> One other piece of data; I have never encountered this error with any
> other GRASS modules... just r.series and t.rast.series.
>
> Thanks,
> Dylan
>
>
>
> On Tue, Oct 13, 2015 at 3:09 AM, Markus Neteler  wrote:
>> Hi Dylan,
>>
>> please post to the list which GRASS GIS version you are using
>> (g.version or wxGUI HELP) and the OS.
>>
>> Best
>> Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

Hi Markus,

GRASS version information:

 ./configure  --without-odbc --without-mysql --with-readline
--with-cxx --enable-largefile --with-freetype
--with-freetype-includes=/usr/include/freetype2 --with-sqlite
--with-python --with-pthread --with-geos=/usr/local/bin/geos-config
--without-opencl --with-opencl-includes=/usr/include/CL/
--with-postgres --with-postgres-includes=/usr/include/postgresql/
--with-postgres-libs=/usr/lib/
--with-proj-share=/usr/local/share/proj/

libgis Revision: 64732
libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)

PROJ.4: 4.8.0
GDAL/OGR: 2.0.0
GEOS: 3.4.2
SQLite: 3.7.9

I find it strange that I have encountered this mysterious error
(occasionally) over the last couple of years while tracking
grass_trunk.

One other piece of data; I have never encountered this error with any
other GRASS modules... just r.series and t.rast.series.

Thanks,
Dylan



On Tue, Oct 13, 2015 at 3:09 AM, Markus Neteler  wrote:
> Hi Dylan,
>
> please post to the list which GRASS GIS version you are using
> (g.version or wxGUI HELP) and the OS.
>
> Best
> Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Tue, Oct 13, 2015 at 11:55 AM, Markus Neteler  wrote:
> On Tue, Oct 13, 2015 at 8:43 PM, Dylan Beaudette
>  wrote:
>> Dangit... This is strange, just did an 'svn up', make distclean, make,
>> make install, and now this:
>>
>> Welcome to GRASS GIS 7.1.svn (r66487)
>> GRASS GIS homepage:  http://grass.osgeo.org
>> This version running through:Bash Shell (/bin/bash)
>> Help is available with the command:  g.manual -i
>> See the licence terms with:  g.version -c
>> Start the GUI with:  g.gui wxpython
>> When ready to quit enter:exit
>>
>> GRASS 7.1.svn (prism):~/src/grass_trunk > g.version -r
>> GRASS 7.1.svn (2015)
>> libgis Revision: 64732
>> libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)
>>
>> Has it been so long since I have compiled GRASS that I have missed
>> something?
>
> Yep:
> https://trac.osgeo.org/grass/changeset/65591
> "Prevent concurrent raster reads when a mask is present"
>
> (which also got backported subsequently)
>
>> I see that the libgis is still "old".
>
> ... this is unrelated since the issue was in r.mapcalc (hence
> affecting all modules using it).
>
> The true revision number of interest is the one next to GRASS GIS
> 7.1.svn (r66487).
>
> I guess your issue is now solved.
>
> Markus

Maybe. I updated my local copy of grass_trunk last Monday, compiled,
and experienced the issues with r.series and t.rast.series... in the
absence of a MASK map.

Are the *.series modules a convenient front-end to r.mapcalc?

Thanks!
Dylan
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Sören Gebbert]

Hi Dylan,
r.series is a module implemented in C with no relations to r.mapcalc.
t.rast.series is a Python module that makes use of r.series internally.

Best regards
Soeren

2015-10-13 21:06 GMT+02:00 Dylan Beaudette :
> On Tue, Oct 13, 2015 at 11:55 AM, Markus Neteler  wrote:
>> On Tue, Oct 13, 2015 at 8:43 PM, Dylan Beaudette
>>  wrote:
>>> Dangit... This is strange, just did an 'svn up', make distclean, make,
>>> make install, and now this:
>>>
>>> Welcome to GRASS GIS 7.1.svn (r66487)
>>> GRASS GIS homepage:  http://grass.osgeo.org
>>> This version running through:Bash Shell (/bin/bash)
>>> Help is available with the command:  g.manual -i
>>> See the licence terms with:  g.version -c
>>> Start the GUI with:  g.gui wxpython
>>> When ready to quit enter:exit
>>>
>>> GRASS 7.1.svn (prism):~/src/grass_trunk > g.version -r
>>> GRASS 7.1.svn (2015)
>>> libgis Revision: 64732
>>> libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)
>>>
>>> Has it been so long since I have compiled GRASS that I have missed
>>> something?
>>
>> Yep:
>> https://trac.osgeo.org/grass/changeset/65591
>> "Prevent concurrent raster reads when a mask is present"
>>
>> (which also got backported subsequently)
>>
>>> I see that the libgis is still "old".
>>
>> ... this is unrelated since the issue was in r.mapcalc (hence
>> affecting all modules using it).
>>
>> The true revision number of interest is the one next to GRASS GIS
>> 7.1.svn (r66487).
>>
>> I guess your issue is now solved.
>>
>> Markus
>
> Maybe. I updated my local copy of grass_trunk last Monday, compiled,
> and experienced the issues with r.series and t.rast.series... in the
> absence of a MASK map.
>
> Are the *.series modules a convenient front-end to r.mapcalc?
>
> Thanks!
> Dylan
> ___
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Markus Neteler]

On Tue, Oct 13, 2015 at 8:59 PM, Dylan Beaudette
 wrote:
...
>>> libgis Revision: 64732

Ah, I read on mobile, not realizing that this was the libgis rev number.

...
> How does the "libgis" revision number relate to the version reported
> when starting GRASS (e.g. SVN revision)?

Not at all...

ok I find this libgis rev number continuously confusing and useless as
already posted some years ago.
In the end only the starting-GRASS-SVN-revision gives me true information.

@devs: I suggest to less prominently advertise the lib*gis* rev number
in g.version.

Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

Dangit... This is strange, just did an 'svn up', make distclean, make,
make install, and now this:

Welcome to GRASS GIS 7.1.svn (r66487)
GRASS GIS homepage:  http://grass.osgeo.org
This version running through:Bash Shell (/bin/bash)
Help is available with the command:  g.manual -i
See the licence terms with:  g.version -c
Start the GUI with:  g.gui wxpython
When ready to quit enter:exit

GRASS 7.1.svn (prism):~/src/grass_trunk > g.version -r
GRASS 7.1.svn (2015)
libgis Revision: 64732
libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)

Has it been so long since I have compiled GRASS that I have missed
something? I see that the libgis is still "old".

?

Dylan






On Tue, Oct 13, 2015 at 11:06 AM, Markus Neteler  wrote:
>
> On Oct 13, 2015 7:01 PM, "Dylan Beaudette" 
> wrote:
>>
>> Hi Markus,
>>
>> GRASS version information:
>>
>>  ./configure  --without-odbc --without-mysql --with-readline
>> --with-cxx --enable-largefile --with-freetype
>> --with-freetype-includes=/usr/include/freetype2 --with-sqlite
>> --with-python --with-pthread
>
> I would not use pthread.
>
>> --with-geos=/usr/local/bin/geos-config
>> --without-opencl --with-opencl-includes=/usr/include/CL/
>> --with-postgres --with-postgres-includes=/usr/include/postgresql/
>> --with-postgres-libs=/usr/lib/
>> --with-proj-share=/usr/local/share/proj/
>>
>> libgis Revision: 64732
>
> This is fairly old. Trunk is r66487.
>
> Can you update?
>
>> libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)
>>
>> PROJ.4: 4.8.0
>> GDAL/OGR: 2.0.0
>> GEOS: 3.4.2
>> SQLite: 3.7.9
>
> Best
> Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

Hi Moritz,

Not using a MASK in this case. Fairly basic work-flow, using "tmin"
and "tmax" (Space Time Raster Dataset) loaded via r.external.


# make a where clause for finding the current year
  wc="strftime('%Y', start_time) = '"$year"'"

  # extract current dataset: "copies" by reference
  t.rast.extract --q --o input=tmin output=tmin_subset
basename=tmin_subset where="$wc"
  t.rast.extract --q --o input=tmax output=tmax_subset
basename=tmax_subset where="$wc"

  # GDD for current year: slow from external rasters
  # NOTE: 2 CPUs so that disk isn't thrashed
  t.rast.mapcalc --o nprocs=2 method=equal
input=tmin_subset,tmax_subset output=gdd basename=gdd expr="(((
min((tmax_subset * 1.8 + 32.0), 86.0) + max((tmin_subset * 1.8 +
32.0), 50) ) / 2) - 50)"


... the error occurs at this step in a non-deterministic pattern.
Strangely enough, the errors are more frequent during the morning
hours vs. afternoon hours!

  # sum GDD for this year: fast from SSD, but only single CPU thread
  t.rast.series --o --q in=gdd out=gdd_$year method=sum


Thanks,
Dylan



On Tue, Oct 13, 2015 at 1:04 AM, Moritz Lennert
 wrote:
> On 12/10/15 23:35, Dylan Beaudette wrote:
>>
>> Hi,
>>
>> Over the last couple of years I have noticed a very strange raster
>> corruption (?) issues when using r.series, and now more recently,
>> t.rast.series. Typically, I'll generate a large number of maps with
>> r.sun or t.rast.mapcalc and then aggregate the series with r.series or
>> t.rast.series. About 50% of the time the command runs as expected, the
>> other half of the time r.series or t.rast.series gives me an error
>> like this:
>>
>> Error reading raster data for row xxx (testmap)
>>
>> After this error, I can no longer perform any kind of operation on map
>> "testmap" without the dreaded Error reading raster data for row xxx...
>>
>> The situation was worse when using a MASK map, possibly related to a
>> similar (fixed?) issue discussed in this thread:
>>
>> http://lists.osgeo.org/pipermail/grass-dev/2015-July/075627.html
>>
>> Within that thread, Glynn mentioned that this type of error was
>> probably related to pthreads and concurrent processes. The temporary
>> fix entailed:
>>
>> export WORKERS=0
>>
>> I have tried this on my machine but the results are the same,
>> non-deterministic corruption (?) of one input to r.series or
>> t.rast.series.
>>
>> I have encountered this error on several disks, mirrored HDDs, single
>> HDD, and now on an SSD. I don't think that this is a disk problem,
>> rather, something that r.series or t.rast.series is "doing" to the
>> files it operates on.
>>
>> Is there some possibility that one of these commands is leaving a file
>> "open" or in some kind of intermediate state that prevents subsequent
>> commands from accessing the file?
>>
>> I'll try to create a sample dataset to send over. In the meantime is
>> there any kind of diagnostic information that I can report back with?
>
>
> Are you using a mask, as was the case in the thread you cite ?
>
> Moritz
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Markus Neteler]

On Oct 13, 2015 7:01 PM, "Dylan Beaudette" 
wrote:
>
> Hi Markus,
>
> GRASS version information:
>
>  ./configure  --without-odbc --without-mysql --with-readline
> --with-cxx --enable-largefile --with-freetype
> --with-freetype-includes=/usr/include/freetype2 --with-sqlite
> --with-python --with-pthread

I would not use pthread.

> --with-geos=/usr/local/bin/geos-config
> --without-opencl --with-opencl-includes=/usr/include/CL/
> --with-postgres --with-postgres-includes=/usr/include/postgresql/
> --with-postgres-libs=/usr/lib/
> --with-proj-share=/usr/local/share/proj/
>
> libgis Revision: 64732

This is fairly old. Trunk is r66487.

Can you update?

> libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)
>
> PROJ.4: 4.8.0
> GDAL/OGR: 2.0.0
> GEOS: 3.4.2
> SQLite: 3.7.9

Best
Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Tue, Oct 13, 2015 at 11:06 AM, Markus Neteler  wrote:
>
> On Oct 13, 2015 7:01 PM, "Dylan Beaudette" 
> wrote:
>>
>> Hi Markus,
>>
>> GRASS version information:
>>
>>  ./configure  --without-odbc --without-mysql --with-readline
>> --with-cxx --enable-largefile --with-freetype
>> --with-freetype-includes=/usr/include/freetype2 --with-sqlite
>> --with-python --with-pthread
>
> I would not use pthread.

OK, good to know. I will re-compile without it and report back.

>> --with-geos=/usr/local/bin/geos-config
>> --without-opencl --with-opencl-includes=/usr/include/CL/
>> --with-postgres --with-postgres-includes=/usr/include/postgresql/
>> --with-postgres-libs=/usr/lib/
>> --with-proj-share=/usr/local/share/proj/
>>
>> libgis Revision: 64732
>
> This is fairly old. Trunk is r66487.
>
> Can you update?

How does the "libgis" revision number relate to the version reported
when starting GRASS (e.g. SVN revision)?


>> libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)
>>
>> PROJ.4: 4.8.0
>> GDAL/OGR: 2.0.0
>> GEOS: 3.4.2
>> SQLite: 3.7.9
>
> Best
> Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Markus Neteler]

On Tue, Oct 13, 2015 at 8:43 PM, Dylan Beaudette
 wrote:
> Dangit... This is strange, just did an 'svn up', make distclean, make,
> make install, and now this:
>
> Welcome to GRASS GIS 7.1.svn (r66487)
> GRASS GIS homepage:  http://grass.osgeo.org
> This version running through:Bash Shell (/bin/bash)
> Help is available with the command:  g.manual -i
> See the licence terms with:  g.version -c
> Start the GUI with:  g.gui wxpython
> When ready to quit enter:exit
>
> GRASS 7.1.svn (prism):~/src/grass_trunk > g.version -r
> GRASS 7.1.svn (2015)
> libgis Revision: 64732
> libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)
>
> Has it been so long since I have compiled GRASS that I have missed
> something?

Yep:
https://trac.osgeo.org/grass/changeset/65591
"Prevent concurrent raster reads when a mask is present"

(which also got backported subsequently)

> I see that the libgis is still "old".

... this is unrelated since the issue was in r.mapcalc (hence
affecting all modules using it).

The true revision number of interest is the one next to GRASS GIS
7.1.svn (r66487).

I guess your issue is now solved.

Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Tue, Oct 13, 2015 at 11:06 AM, Markus Neteler  wrote:
>
> On Oct 13, 2015 7:01 PM, "Dylan Beaudette" 
> wrote:
>>
>> Hi Markus,
>>
>> GRASS version information:
>>
>>  ./configure  --without-odbc --without-mysql --with-readline
>> --with-cxx --enable-largefile --with-freetype
>> --with-freetype-includes=/usr/include/freetype2 --with-sqlite
>> --with-python --with-pthread
>
> I would not use pthread.
>

Hi Markus,

I have done some tests after re-compiling _without_ pthreads. So far,
no errors from r.series. Also I seem to be getting better performance
form r.mapcalc and t.rast.series, especially when working with files
loaded via r.external. I don't know enough about pthreads to speculate
further. Any ideas?

I'll report back when the script finishes, probably 20 hours or so.

Cheers,
Dylan
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

Thank you for the clarification Sören.

Any ideas on how r.series could be leaving maps open or otherwise
corrupting inputs in those cases where:

1. there are a lot of maps (>100)
2. method=sum

Dylan

On Tue, Oct 13, 2015 at 12:11 PM, Sören Gebbert
 wrote:
> Hi Dylan,
> r.series is a module implemented in C with no relations to r.mapcalc.
> t.rast.series is a Python module that makes use of r.series internally.
>
> Best regards
> Soeren
>
> 2015-10-13 21:06 GMT+02:00 Dylan Beaudette :
>> On Tue, Oct 13, 2015 at 11:55 AM, Markus Neteler  wrote:
>>> On Tue, Oct 13, 2015 at 8:43 PM, Dylan Beaudette
>>>  wrote:
 Dangit... This is strange, just did an 'svn up', make distclean, make,
 make install, and now this:

 Welcome to GRASS GIS 7.1.svn (r66487)
 GRASS GIS homepage:  http://grass.osgeo.org
 This version running through:Bash Shell (/bin/bash)
 Help is available with the command:  g.manual -i
 See the licence terms with:  g.version -c
 Start the GUI with:  g.gui wxpython
 When ready to quit enter:exit

 GRASS 7.1.svn (prism):~/src/grass_trunk > g.version -r
 GRASS 7.1.svn (2015)
 libgis Revision: 64732
 libgis Date: 2015-02-24 16:54:05 -0800 (Tue, 24 Feb 2015)

 Has it been so long since I have compiled GRASS that I have missed
 something?
>>>
>>> Yep:
>>> https://trac.osgeo.org/grass/changeset/65591
>>> "Prevent concurrent raster reads when a mask is present"
>>>
>>> (which also got backported subsequently)
>>>
 I see that the libgis is still "old".
>>>
>>> ... this is unrelated since the issue was in r.mapcalc (hence
>>> affecting all modules using it).
>>>
>>> The true revision number of interest is the one next to GRASS GIS
>>> 7.1.svn (r66487).
>>>
>>> I guess your issue is now solved.
>>>
>>> Markus
>>
>> Maybe. I updated my local copy of grass_trunk last Monday, compiled,
>> and experienced the issues with r.series and t.rast.series... in the
>> absence of a MASK map.
>>
>> Are the *.series modules a convenient front-end to r.mapcalc?
>>
>> Thanks!
>> Dylan
>> ___
>> grass-dev mailing list
>> grass-dev@lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/grass-dev
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

Hi,

Over the last couple of years I have noticed a very strange raster
corruption (?) issues when using r.series, and now more recently,
t.rast.series. Typically, I'll generate a large number of maps with
r.sun or t.rast.mapcalc and then aggregate the series with r.series or
t.rast.series. About 50% of the time the command runs as expected, the
other half of the time r.series or t.rast.series gives me an error
like this:

Error reading raster data for row xxx (testmap)

After this error, I can no longer perform any kind of operation on map
"testmap" without the dreaded Error reading raster data for row xxx...

The situation was worse when using a MASK map, possibly related to a
similar (fixed?) issue discussed in this thread:

http://lists.osgeo.org/pipermail/grass-dev/2015-July/075627.html

Within that thread, Glynn mentioned that this type of error was
probably related to pthreads and concurrent processes. The temporary
fix entailed:

export WORKERS=0

I have tried this on my machine but the results are the same,
non-deterministic corruption (?) of one input to r.series or
t.rast.series.

I have encountered this error on several disks, mirrored HDDs, single
HDD, and now on an SSD. I don't think that this is a disk problem,
rather, something that r.series or t.rast.series is "doing" to the
files it operates on.

Is there some possibility that one of these commands is leaving a file
"open" or in some kind of intermediate state that prevents subsequent
commands from accessing the file?

I'll try to create a sample dataset to send over. In the meantime is
there any kind of diagnostic information that I can report back with?

Thanks,
Dylan
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev