Re: [GRASS-dev] PSDriver
Brad ReDacted wrote: > Can someone tell me what spec the (enhanced) postscript driver > (lib/psdriver) adheres to? I am having difficulty looking up one of the > parameters (box erase, in particular - erase.c). > > I found references to PostScript levels, but no mention of EPS, which is > clearly being used in this case. AFAICT, there is no such box erase in EPS. If the output filename ends in ".eps", then the "ps.encapsulated" flag is set. This does two things: PS_Set_window() writes "EPSWINDOW" instead of "WINDOW", and PS_Erase() writes "ERASE" instead of a "BOX" command. All commands are defined in psdriver.ps. The PS driver doesn't attempt to generate true Encapsulated PostScript, it just tries to be slightly more useful if you're going to something other than send the file straight to a printer. Essentially, if ps.encapsulated is set, the driver avoids issuing "destructive" commands. BOX just draws a filled rectangle, while ERASE executes an "erasepage" command. EPSWINDOW is identical to WINDOW except the former doesn't execute "initclip". I have no idea how much use (if any) the PS driver has. Its main advantage over using the cairo driver is that the PS driver doesn't rasterise vector commands, whereas cairo will do this for features which can't otherwise be implemented (e.g. alpha blending). This can result in PS files generated by the cairo driver being much larger than those generated by the PS driver while having worse image quality. ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Reasons for GDAL dynamic library load in code?
Vaclav Petras wrote: > To be sure removing it completely is a good step, I wanted to check the > original motivation. Any ideas? The commit message doesn't mention the > motivation and I'm not getting anything from Internet searches. Dependencies. GDAL has lots of them. If libraster is linked directly against GDAL and one of the GDAL dependencies is missing or broken, GRASS will be almost completely non-functional. By using dlopen(), if there's an issue with GDAL it only matters if you try to use GDAL-linked (r.external) maps. This isn't an issue if you're getting GRASS (and everything it depends upon) via your distribution's package manager. It may be an issue for DIY builds. ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Question regarding GRASS_NOTIFY
Nicklas Larsson via grass-dev wrote: > I personally never had the need for the use of GRASS in this way, so forgive > my ignorance in this regard. > However, one thing stands out very clear from my newly gained experience: > there is a lack of documentation on this use of GRASS_NOTIFY. For instance, > it is not clear to me is whether it is required or optional to set > GRASS_NOTIFY > with e.g.: > > export "GRASS_NOTIFY=kill -USR1 `pidof ximgview`” > > for this to work as intended. Surely it is not expected a user should look > for this > information in the mailing list. I think that it's "expected" that the user will just use the GUI. It seems doubtful that anyone (other than me) actually used this. I no longer actively follow GRASS; I only remained subscribed to the list for cases (like this) where someone is asking about arcane details of stuff I worked on in the past. > > What sort of "sanitisation" would you suggest? The variable is set by > > the user, its value is passed directly to the shell. > > > I’d say it would be better to avoid sanitation, with what I mean making sure > GRASS_NOTIFY hasn’t been compromised, at all. Couldn’t the desired outcome > of using GRASS_NOTIFY be implemented in another way? Well, it could be made a GRASS ($GISRC) variable. A fixed command name could be inconvenient if you have multiple GRASS sessions in different shells (again, I don't think this is "typical" user behaviour). It could use some other mechanism entirely (e.g. a filename identifying a socket or named pipe to which D_close_driver would write) would work, but would be a non-trivial effort for something that probably isn't even used. I'm not sure manipulating GRASS_NOTIFY gets you much compared to manipulating e.g. EDITOR or BROWSER. And unless there's been a significant effort on security since I was active, using GRASS with untrusted inputs or an untrusted environment has much bigger issues. In any case, ximgview/wxpyimgview don't appear to have had any substantial changes or issues (i.e. nothing that doesn't appear to be a repository-wide clean-up) in the last decade, so non-GUI usage of the display subsystem is probably a non-issue at this point. ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Question regarding GRASS_NOTIFY
Nicklas Larsson via grass-dev wrote: > Is there anyone out there who know about usage of the environment > variable GRASS_NOTIFY, historically and in the present? It's an external interface; it isn't used internally. The intention is described in the email you cited: to allow a program (specifically, ximgview or similar) to be notified of any changes to the image file generated by the display library. It isn't required by the GUI because the GUI can assume that the image will only be modified by d.* commands which the GUI itself runs, so it can refresh the view upon their completion. The idea was to support the use of d.* commands from a shell, providing something similar to the "monitor" system that was being replaced at the time. I've have no idea whether such usage is still common or even officially supported (but scripts/wxpyimgview is still there). > In addressing an otherwise trivial compiler warning, I stumbled into > this for me strange piece of code and usage of GRASS_NOTIFY in the > Raster Display Library. I put up a PR [1] suggesting the removal of > this, but I agree with Vaclav I’d better ask here on the ML. > > The only use of GRASS_NOTIFY is in “D_close_driver()” [2] and the > value of that env variable (if set) is directly inserted in a > “system()” call, without any sanitation! What sort of "sanitisation" would you suggest? The variable is set by the user, its value is passed directly to the shell. ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.series method=minimum yields float map when integer maps are used as input
Veronica Andreo wrote: > If I use a series of integer type maps as input to t.rast.series or > r.series and select minimum as method, the output is a floating point map. > I would have thought that if the input are integers then output should be > integer as well. I realized when I made a plot of the map and got a smooth > legend instead of integer numbers. > Is this an expected behaviour? The type of the output map is independent of the inputs, determined solely by the method. The count, diversity, min_raster and max_raster methods generate integer maps; all other methods generate floating-point maps. It probably wouldn't be particularly involved to add another flag to the method table to indicate that a particular method will always produce an integer result for integer inputs. Also, the use of weights= forces the output to floating-point even if the method doesn't actually support the use of weights. -- Glynn Clements ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] pygrass: How to allow Ctrl-C to stop all subprocesses of GridModule
Moritz Lennert wrote: > Any hint how to allow Ctrl-C to stop all subprocesses ? If you can get their PIDs, you should be able to use os.setpgid() to move them into the same process group as the main script. Ctrl-C (by default) sends SIGINT to all processes in the foreground process group. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.series with file option - too many files open
Markus Metz wrote: > Use the -z flag to analyze large amount of raster maps without hitting open > files limit Or alternatively, see if you can increase the limit. "ulimit -n" displays the soft limit, "ulimit -Hn" displays the hard limit. If they aren't the same, you can use "ulimit -n " to increase the soft limit up to the hard limit (on my system, the soft limit is 1024, the hard limit is 4096). If you're at the hard limit, you can ask your sysadmin to increase it, typically via /etc/security/limits.conf. Increasing the limit won't have the performance cost of -z. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Python 3 porting and unicode
Vaclav Petras wrote:
> * There is no way around the unicode when using Python 3. Unicode is
> inherent part of the language even things such as os.environ or
> sys.stdout.write() work only with unicode. I'm not sure what exactly the
> rule is here, but it seems to be everywhere.
Python 3 has os.environb on Unix. You can use the .detach() method on
text streams to get the underlying binary stream.
> * In relation to the previous point, one of the reasons why unicode is used
> that thinks like text[:10] actually return 10 characters to display.
Although some of those characters may be combining characters or
control codes. Unicode characters don't necessarily map 1:1 with
glyphs.
> * Users of the Python API who are using Python 3 will expect unicode
> strings to work, i.e. expect run_command('g.region', flags='p') to work
> (not just run_command(b'g.region', flags=b'p')).
Even if you automatically encode unicode strings, there's no guarantee
that it will work (e.g. if the string is a filename, then the encoded
string must produce the correct sequence of bytes).
I can't think of any significant cases where it's likely to be
necessary to pass "binary" data via arguments, although it should be
trivial to simply accept data which is already a byte string.
The bigger issue is with output: the output from GRASS commands isn't
guaranteed to be in the locale's encoding (if it's extracted from a
file, it's going to be in whatever encoding the file uses). Returning
bytes allows the user to deal with this; automatically decoding the
data will either raise an exception or return mojibake if the encoding
doesn't match.
> * It seems hard to predict when we will know the right encoding of the
> text.
Which is why byte-oriented interfaces still exist and still matter,
and will do so for the foreseeable future.
Python's solution is to accelerate standardisation on Unicode by
making the alternatives as painful as possible. Yet legacy encodings
remain widespread
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] scientific notation in r.mapcalc ?
Markus Neteler wrote:
> > I just stumbled upon this while running a large model, and noticed that I
> > couldn't easily find an answer:
> >
> > IIUC r.mapcalc does not support scientific notation of floating point
> > numbers (i.e. 2.54e-05 instead of 0.254). Is that correct ?
> >
> > How difficult would it be to implement the support of such notation ?
>
> We just stumbled over the same issue... would be nice to have.
It works here:
$ r.mapcalc "foo = 2.54e-05"
$ r.info -r foo
min=2.54e-05
max=2.54e-05
$ g.version -gr
version=7.3.svn
date=2016
revision=r70088
build_date=2017-06-16
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
libgis_revision=70829
libgis_date="2017-04-04 08:43:02 +0100 (Tue, 04 Apr 2017) "
mapcalc.l specifically recognises exponential notation:
I [0-9]+
E [eE][-+]?[0-9]+
%%
{I}"."{I}?{E}? |
"."{I}{E}? {
yylval.fval = atof(yytext);
return DOUBLE;
}
atof() is specified as:
7.20.1.1 The atof function
...
[#2] The atof function converts the initial portion of the
string pointed to by nptr to double representation. Except
for the behavior on error, it is equivalent to
strtod(nptr, (char **)NULL)
and strtod() as
7.20.1.3 The strtod, strtof, and strtold functions
...
[#3] The expected form of the subject sequence is an
optional plus or minus sign, then one of the following:
-- a nonempty sequence of decimal digits optionally
containing a decimal-point character, then an optional
exponent part as defined in 6.4.4.2;
I note that any locale-specific varations are supposed to be in
addition to the format used by the "C" locale, so even if LC_NUMERIC
gets set (and it shouldn't), that shouldn't affect it.
Is this issue a wxGUI thing?
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] how to reproject a raster map with absolute numbers without losing data
Moritz Lennert wrote: > > For anything other than lat-lon locations, G_area_of_cell_at_row() > > assumes that the area of a cell is constant over the region. That's > > true for equal-area projections, and not far off for small scales > > (where projection would be almost an affine transformation), but it > > could be quite far off at larger scales. > > Do I understand correctly that you would like the function to return > geodesic area in projected locations? That would be a nice feature, > but I wouldn't make it the default. I think most people would expect > constant pixel area in such location, knowing (at least some) that > there is inherent error in the projection. I'm not suggesting that G_area_of_cell_at_row() should be changed, nor the new r.mapcalc area() function. Calculating the actual area based upon the projection is computationally expensive.and rarely necessary. I'm just suggesting that it might be useful if there was some way to get that information, other than by creating maps containing latitude and longitude values, projecting those with r.proj, then using r.mapcalc to calculate the area (or other properties) of each cell based upon those. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] how to reproject a raster map with absolute numbers without losing data
Moritz Lennert wrote: > Yes. Do we have any existing module to calculate area by pixel ? An > area() function in r.mapcalc would be nice... I notice that this has been added, but I'm not sure that it's adequate for this task. For anything other than lat-lon locations, G_area_of_cell_at_row() assumes that the area of a cell is constant over the region. That's true for equal-area projections, and not far off for small scales (where projection would be almost an affine transformation), but it could be quite far off at larger scales. Ideally, r.proj should provide the option to generate maps containing metadata for a projection: source coordinates, horizontal/vertical scale factors, area scale factor, rotation angle, skew angle. I.e. projected coordinates, horizontal/vertical first derivatives, and information derived from them (maybe also second derivatives, and derived quantaties such as curvature or radius of curvature, but that can be quite inaccurate for discrete samples). IOW, rather than inverse-projecting coordinates and performing a look-up in a source map, it would inverse-project coordinates and write the resulting coordinates (or differences). The main issue is an appropriate choice of delta. Too large a value means that you're calculating an average over a non-negligible distance, too small a value means that you lose accuracy due to subtracting numerically-close values. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] does GRASS need HOME to be defined?
Helmut Kudrnovsky wrote:
> regarding windows, the configuration directory is defined by the %APPDATA%
> variable; HOME is defined by %HOMEDRIVE% and %HOMEPATH%.
>
> it expands to:
>
> >>> import os
> >>> os.getenv('HOME')
> 'C:\\Users\\yourusername'
> >>> os.getenv('APPDATA')
> 'C:\\Users\\yourusername\\AppData\\Roaming'
>
> would it harm if %HOME% would be extended from _C:\Users\yourusername\_ to
> _C:\Users\yourusername\Documents_ in windows?
Note that:
1. %HOMEPATH% isn't guaranteed to be Users\.
2. The preferred way of referencing that directory is %USERPROFILE%.
3. While %USERPROFILE%\Documents is created automatically, that isn't
guaranteed to be the location of the user's "My Documents" folder (it
can be changed, and in networked environments is typically located on
a file server).
FWIW, the "My Documents" location can be obtained by querying the
registry key
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell
Folders\Personal
then expanding the result; the default value is "%USERPROFILE\Documents".
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] does GRASS need HOME to be defined?
Helmut Kudrnovsky wrote: > >> anyone more insight, if %HOME% is needed in winGRASS? > > > >Done by Glynn in: > >https://trac.osgeo.org/grass/changeset/37873 > > does GRASS need HOME to be defined? > > any hints? It's used G__home() on Unix (and as a fallback on Windows) and it's used for G_rc_path() on Windows (for some reason, the Unix version uses getpwuid(), which is probably incorrect in this context). It may also be used implicitly by library functions or external programs looking for configuration files. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] What is considered as valid location and valid mapset
Vaclav Petras wrote: > I wonder what is a valid location and what is a valid mapset. The > lib/init/grass.py file says: > > def is_mapset_valid(full_mapset): > return os.access(os.path.join(full_mapset, "WIND"), os.R_OK) > > def is_location_valid(gisdbase, location): > return os.access(os.path.join(gisdbase, location, "PERMANENT", > "DEFAULT_WIND"), os.F_OK) > > This is what I've done based on the code which was there before > refactoring. I wonder if it is correct. If I need to switch to a mapset, I > want the region to be editable, so I need os.W_OK in that case. Valid for use as a source of maps and valid for use as the current mapset are different things. For use as the current mapset, the entire directory hierarchy should be owned by the current user. G_mapset_permissions() requires that the mapset directory is owned by the current user unless you're on Windows or libgis was built with SKIP_MAPSET_OWN_CHK defined and the environment variable GRASS_SKIP_MAPSET_OWNER_CHECK is set to a non-empty string. Group-writable mapsets are asking for trouble (it's quite likely that users will end up creating maps which the mapset's owner can't remove). > On the other hand, is WIND file needed? Isn't it created > automatically from DEFAULT_WIND? No. It's created from DEFAULT_WIND by "g.region -d" and functions or modules which create a new mapset. Most modules will fail if WIND doesn't exist (assuming that neither GRASS_REGION nor WIND_OVERRIDE are set). > And further, is DEFAULT_WIND the right requirement for a valid PERMANENT > mapset? Are some PROJ_* files needed? Ideally, neither DEFAULT_WIND nor PROJ_INFO should be required until you do something that actually uses them. The check is just a heuristic; a directory containing a PERMANENT/DEFAULT_WIND file is probably a GRASS location, while a directory lacking it probably isn't. I'm not sure whether X-Y locations would even have a PROJ_INFO file. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] G70: wximgview compilation issue and fix
Markus Neteler wrote: > when compiling GRASS GOS 7.0.svn on Fedora 24, I get > > neteler@oboe wximgview]$ make > : && c++ -L/home/neteler/software/grass70/dist.x86_64-pc-linux-gnu/lib > -L/home/neteler/software/grass70/dist.x86_64-pc-linux-gnu/lib > -Wl,--no-undefined -Wl,-z,now -Wl,--export-dynamic > -Wl,-rpath-link,/home/neteler/software/grass70/dist.x86_64-pc-linux-gnu/lib > -o /home/neteler/software/grass70/dist.x86_64-pc-linux-gnu/bin/wximgview > OBJ.x86_64-pc-linux-gnu/main.o-lgrass_gis.7.0.5svn -pthread > -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld > -lwx_gtk2u_richtext-2.8 -lwx_gtk2u_aui-2.8 -lwx_gtk2u_xrc-2.8 > -lwx_gtk2u_qa-2.8 -lwx_gtk2u_html-2.8 -lwx_gtk2u_adv-2.8 > -lwx_gtk2u_core-2.8 -lwx_baseu_xml-2.8 -lwx_baseu_net-2.8 > -lwx_baseu-2.8-lm > /usr/bin/ld: OBJ.x86_64-pc-linux-gnu/main.o: relocation R_X86_64_32 > against `.rodata' can not be used when making a shared object; > recompile with -fPIC But it's not trying to make a shared object. My guess is that it's related to -Wl,-z,relro, possibly an interaction between that flag and either C++ or the wxWidgets libraries. > This patch solves it: > -EXTRA_CFLAGS = $(WXWIDGETSCXXFLAGS) -DWXINTL_NO_GETTEXT_MACRO > +EXTRA_CFLAGS = $(WXWIDGETSCXXFLAGS) -DWXINTL_NO_GETTEXT_MACRO -fPIC > but I am not sure if it is the right solution to be applied to SVN. > > Any ideas? Executables shouldn't require -fPIC. It may be that we need separate LINK_FLAGS variables for C and C++ programs. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] How to: if undefined, set to NULL?
Vaclav Petras wrote:
> > Updated in sandbox, see: https://trac.osgeo.org/grass/changeset/69365
> >
> > Is it though correct?
> >
>
> I was not looking at the logic but
>
> if (hue == -1.0) {
>
> is not safe. Although it will probably work most of the time if you set it
> to the exactly same value (`hue = -1;`),
There's no "probably" about it. If you execute "hue = -1;", a
subsequent "if (hue == -1)" comparison *will* succeed.
Rounding error only occurs when the correct result isn't exactly
representable, which isn't the case here. It doesn't occur randomly.
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] where is GRASS looking for these dependencies?
Michael Barton wrote: > Can someone familiar with the GRASS build system tell me where it is > looking for the following items in the configure string when > compiling? > > --with-x > --with-cxx > --with-opengl=aqua > --with-python > --with-opencl --with-x is built into AC_PATH_XTRA which is a standard autoconf macro. The others originate with a LOC_ARG_WITH macro (which is a shallow wrapper around AC_ARG_WITH). This sets the variable with_foo to "yes" if --with-foo is used with no argument, to "no" if --without-foo is used, to the argument if --with-foo is used with an argument, and to the default value (the third argument of LOC_ARG_WITH, defaulting to "no") if --without-foo isn't used. > Thanks. I’m trying to find out if any of these are hardwired to > specifically look in /usr/.. for stuff. Not by configure.in. AC_PATH_XTRA may have some hardwired defaults, some of the other tests use pkgconf by default. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] font size in d.legend
Paulo van Breugel wrote: > I am creating some maps (png format) using the GRASS_RENDER_IMMEDIATE=cairo > and d.* calls. I am setting width and height based on a final desired > 300dpi for the output image (i.e., pixels = # inch x 300). > > However, the font size as can be set in e.g., d.legend is, I assume, based > on the assumption that the output image has a DPI of 72. Is there any way > to set the DPI of the output image? For the cairo driver, I believe that the font size is the size of 1-em in pixels (the cairo documentation says "user space units"; in the absence of any additional transformations, I believe that this means pixels). -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] cProfile makes TypeError
Eva Stopková wrote:
> I would like to ask for advice. I am trying to make a visualization of the
> script process using cProfile according to [1], but it gives me an error:
>
> File
> "~/grass7/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/ctypes_loader.py",
> line 187, in _create_ld_so_cache
> for path in glob.glob("%s/*.s[ol]*" % dir):
> TypeError: __init__() takes exactly 4 arguments (2 given)
This seems to suggest that it's picking up some other "glob" module,
rather than the one in the standard library.
Does running the script without using cProfile work?
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] NULL file compression by default
Markus Neteler wrote: > according to the current documentation, the NULL files are still > uncompressed [1]. > Is there an important reason to not switch it on by default? The fact that no "release" version supports reading compressed null files. So anyone not explicitly disabling compression will be creating maps which cannot be read by anyone using e.g. 7.0.4. First, put out a release version which supports reading compressed null files. Once it's been out long enough for adoption to be widespread, you can consider compressing null files by default. Until then, compression needs to be a conscious decision. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] hacking configure for gettext
Michael Barton wrote:
> I would like GRASS to use gettext that is installed in a location
> NOT in the standard /usr/local... I have no problem building gettext
> in such a location but I am having problems telling GRASS where to
> find it.
>
> The configure switch to enable gettext is --with-nls. I've tried
> adding additional arguments
>
> --with-nls-includes="/Users/cmbarton/grass_source/gettext/gettext_dist/include"
> --with-nls-libs="/Users/cmbarton/grass_source/gettext/gettext_dist/lib"
>
> to tell GRASS where to find gettext but these do not seem to do the
> trick.
>
> Can someone familiar with the build system suggest how to specify
> where to find gettext or now to modify configure so that GRASS can
> find gettext in this other location?
Currently, gettext is treated as a system function, i.e. there's no
way to configure search paths beyond the generic --with-includes= and
--with-libs= switches. It's checked initially without additional
libraries (AC_CHECK_FUNC), and if that fails with -lintl
(AC_CHECK_LIB).
To treat it as an external library function, try the attached patch.
Note that you'll need to add $(INTLINC) to ... something. probably
Either INC (in Grass.make) or NLS_CFLAGS (in Compile.make). Adding it
to individual Makefiles isn't feasible because is included
from , which is included by practically everything.
--
Glynn Clements <gl...@gclements.plus.com>
Index: configure.in
===
--- configure.in(revision 68639)
+++ configure.in(working copy)
@@ -321,6 +321,9 @@
LOC_ARG_WITH_INC(readline, Readline)
LOC_ARG_WITH_LIB(readline, Readline)
+LOC_ARG_WITH_INC(nls, NLS)
+LOC_ARG_WITH_LIB(nls, NLS)
+
LOC_ARG_WITH_INC(tiff, TIFF)
LOC_ARG_WITH_LIB(tiff, TIFF)
@@ -1652,20 +1655,28 @@
LOC_CHECK_USE(nls,NLS,USE_NLS)
+if test -n "${USE_NLS}" ; then
+AC_DEFINE(USE_NLS)
+
+INTLINC=
+
+LOC_CHECK_INC_PATH(nls,NLS,INTLINC)
+
+LOC_CHECK_INCLUDES(libintl.h,NSL,$INTLINC)
+
INTLLIB=
HAVE_NLS=
-if test -n "${USE_NLS}" ; then
-AC_DEFINE(USE_NLS)
+LOC_CHECK_LIB_PATH(nsl,NLS,INTLLIB)
-AC_CHECK_FUNC(gettext, INTLLIB=, [
-AC_CHECK_LIB(intl, gettext, INTLLIB=-lintl, [
-AC_MSG_ERROR([*** Unable to locate gettext() function.])
-])])
+LOC_CHECK_FUNC(gettext,NLS,INTLLIB,[
+LOC_CHECK_LIBS(intl,gettext,NLS,$INTLLIB,INTLLIB,,,)
+])
HAVE_NLS=1
fi
+AC_SUBST(INTLINC)
AC_SUBST(INTLLIB)
AC_SUBST(HAVE_NLS)
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.in.gdal: add scale and shift parameters?
Markus Neteler wrote: > sometimes it is convenient to scale/shift input maps on the fly during > the import in order to avoid an extra step with r.mapcalc (think: tons > of scaled Landsat data etc). Perhaps a better option (albeit one which requires some amount of design, rathern than just coding) would be to have this performed during read, i.e. similarly to how reclass maps are handled. This would allow it to be used with r.external maps without needing to make a copy (which largely defeats the point of using r.external). It would also allow most maps to be stored as CELL, with better precision than FCELL (32 bits rather than 24) and less space than DCELL. It's quite rare for physical measurements to be sufficiently accurate that 32 bits isn't enough. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] start supporting python3 for GRASS
Pietro wrote:
> I committed some changes to start supporting python3 (r68348 - r68367)
Regarding r68357 ("fix in python3 sys.stdout.write accept only strings not
bytes"):
if not lines or lines[0] != b"@ARGS_PARSED@":
-sys.stdout.write(s)
+stdout = os.fdopen(sys.stdout.fileno(), 'wb')
+stdout.write(s)
You can get the underlying byte stream for a text stream using the
.buffer property (i.e. sys.stdout.buffer.write()).
OTOH, this probably doesn't matter much here, as sys.stdout will
almost certainly be a real file (i.e. .fileno() will be valid).
For the script module, it might be better to just replace the standard
streams with their underlying byte streams, i.e.
sys.stdin = sys.stdin.detach()
sys.stdout = sys.stdout.detach()
sys.stderr = sys.stderr.detach()
[The .detach() method returns the underlying byte stream *after*
detaching it from the TextIOWrapper, to avoid any issues arising from
attempting to use both.]
Anything using the script module should be using byte strings
throughout (in spite of how awkward Python 3 tries to make this).
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] grass.info versus print
Vaclav Petras wrote: > Perhaps I was not completely clear before. I agree with Glynn. > grass.script.message() etc. for messages. print()/sys.stdout for text > output (e.g. tables). In short: g.message for "messages", print or sys.stdout.write() for "data". The reason why Unix has stdout and stderr is so that you can redirect stdout to a file which having stderr associated with the terminal. So messages intended for the user don't interfere with parsing the data, and redirecting the data to the file doesn't hide messages from the user. So, assuming that the two streams will be separated should provide some clues as to which is appropriate for any given case. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] creating temporary mapsets in a parallelized python script
Pietro wrote: > Perhaps we could decorate the function with contextmanager: That's an option, but it complicates matters in the (probably more common) case where you just want to use a single temporary region for the entire script. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] creating temporary mapsets in a parallelized python script
Vaclav Petras wrote:
> > Does this function mean that: while concurrent r.basin runs on different
> > CPUs in the same mapset, there is no interefering by region settings change
> > interefering?
>
> Setting WIND_OVERRIDE (environmental variable) will happen for the current
> process, so it won't affect other processes only subprocess which is
> desired. (In your case, this won't work if you would use this function
> outside of r.basin.) So yes, parallel processes (including g.region) won't
> see WIND_OVERRIDE, so they will see and use the original computational
> region.
The environment will be inherited by any child processes, so a
top-level script can use the same approach (but not use_temp_region()
itself) to use a different named region for each subprocess.
The main issue with use_temp_region() is that the clean-up function
uses the current value of the environment variable to determine which
region to delete.
This could be fixed by passing a lambda to atexit.register(), e.g.
def use_temp_region():
name = "tmp.%s.%d" % (os.path.basename(sys.argv[0]), os.getpid())
run_command("g.region", save=name, overwrite=True)
os.environ['WIND_OVERRIDE'] = name
atexit.register(lambda: run_command("g.remove", flags='f', quiet=True,
type='region', name=name))
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] grass.info versus print
Vaclav Petras wrote: > sys.stdout/print is the way. grass.script.info and others are for messages, > not output, and they go to stderr which might cause mixing of order with > stdout in GUI and I unfortunately don't know how to avoid it. grass.script.info() executes "g.message -i", which calls G_important_message(), which writes to stderr. In general, scripts should use the g.message wrappers rather than Python's "print" statement or os.write(), as that will honour any configuration options (GRASS_VERBOSE, GRASS_MESSAGE_FORMAT, --quiet/--verbose, etc). -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] usage of os.linesep (was: Re: [GRASS-SVN] r68240 - grass/trunk/scripts/v.report_
> It seems to me that one of these is need and simple os.linesep is not
> enough. I'm never sure how Python solves this for splitlines() and others.
str.splitlines() treats any of '\n', '\r' and '\r\n' as a separator,
independent of platform.
The same approach is used if a file is opened with universal newlines
('U' in the mode string).
> I was quite struggling with it in the gunittest where sample data and code
> usually have \n but on Windows you get \r\n in the created data. Now I'm
> usually replacing \r\n everywhere and doing comparisons afterwards but
> that's not really applicable for the normal code.
Unless 'b' is added to the mode, on Windows '\n' will be converted to '\r\n' on
output and '\r\n' to '\n' on input.
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] raster query/r.what error with a r.external-linked raster
Helmut Kudrnovsky wrote: > Program received signal SIGSEGV, Segmentation fault. > 0x74418090 in vtable for __cxxabiv1::__si_class_type_info () >from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > > (gdb) bt > #0 0x74418090 in vtable for __cxxabiv1::__si_class_type_info () >from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > #1 0x7fffed6778e3 in GDALRegister_VICAR () at vicardataset.cpp:824 > #2 0x7fffed494ded in GDALAllRegister () at gdalallregister.cpp:234 This may be an issue with GDAL itself, e.g. an external library having been updated without rebuilding GDAL. If that's the case, any program which calls GDALAllRegister() (e.g. "gdalinfo --formats") would exhibit the same issue. Alternatively, it may be an issue with GDAL and GRASS using different versions of a common library (e.g. libstdc++). If you've updated any system libraries, perform a full re-compile of GRASS (after "make clean") and ensure that GRASS isn't built against a "private" version of any system library used by GDAL. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] os.remove
Paulo van Breugel wrote: > > In the addon r.forestfrag (python script), I used os.remove() to > > remove some temporary files (in lines 230, 258 and 430). However, this > > fails in Windows, with an error message like below: > > > > WindowsError: [Error 32] The process cannot access the file because it > > is being used by another process: > > 'c:\\users\\uqdshana\\appdata\\local\\temp\\tmpwlv54l' > > > > Removing these lines with os.remove() solves the problem, but than the > > user is left with these temporary files. Is there a Window compatible > > way to remove them in the script? > > Perhaps I should have looked harder first, the underlying problem > (described here: https://www.logilab.org/blogentry/17873) seems to be > that the file needs to be closed at OS level, which can be done using > os.close(). Ideally, files should be handled using context managers and "with". Python's own file objects are already context managers, so you can use e.g. with open(filename) as f: data = f.read() # by this point, f has already been closed For dealing with OS-level descriptors, you can use contextlib (Python 2.5+) to easily create context managers, e.g. import os import tempfile from contextlib import contextmanager @contextmanager def temp(): fd, filename = tempfile.mkstemp() try: yield (fd, filename) finally: os.close(fd) os.remove(filename) if __name__ == '__main__': ... with temp() as (fd, filename): whatever() # fd has been closed and the file removed Use of "with" ensures that clean-up happens in the event of an exception or other forms of early exit (e.g. return, break, continue). -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [GRASS-SVN] r68077 - in grass/branches/releasebranch_7_0/scripts: r.blend r.rgb
Vaclav Petras wrote: > I know we have dot in map names in lot of places but I though we want to > advertise underscore now because it works with vectors and database tables > (as well as columns). Maybe if people keep using a dot, someone will eventually care enough about the broken 5.3 vector changes to fix the issue, rather than expecting everyone else to work around it. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.rgb: syntax error, unexpected $end, expecting VARNAME or NAME or STRING
Vaclav Petras wrote:
> > ok. But another error occurs (missing error handling if the output
> > names were selected to be identica):
> > ...
> > Perhaps there is an elegant solution to catch the (user) error above?
>
> Isn't this an issue for every module? I'm not sure if we have a general
> solution (as in the previous case).
Most modules won't explicitly complain if you use the same map name
for multiple outputs, although it's anyone's guess as to what actually
ends up in the map.
I'm reasonably sure that the primary files (cellhd, cell/fcell, null,
f_{format,quant,range} etc) will be consistent, but metadata files
which are written separately (colour table, categories, history, etc)
may be from a different output.
The fact that r.rgb.py fails in this case is really just a
coincidence. It won't complain about using the same name for all
outputs, only for the case where one of the outputs matches the input.
The error arises because, in r.mapcalc, using a name on the left-hand
side of an assignment causes the name to be interpreted as a variable
rather than as a map, and the r#/g#/b# modifiers can only be applied
to maps, not variables.
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] PEP 394 - The "python" Command on Unix-Like Systems
Maciej Sieczka wrote: > What are the prospects, if any, for GRASS Python code to conform to > https://www.python.org/dev/peps/pep-0394 ? > > Not a huge issue to me personally. I'm just curious about GRASS team > opinions on this matter. It might be worth considering. At least in the > future, when more distros move to Python 3. For now, this is best treated as a packaging issue, i.e. people making binary packages for specific distributions should perform any fixup required to make the package conform to the distribution's requirements and conventions. If this starts to become more common, then I'd suggest adding a INSTALL_PYTHON variable to the build system which can be used similarly to $(INSTALL) or $(INSTALL_DATA), which would replace the shebang as part of the copying process. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] python manuals pages
Vaclav Petras wrote: > > > We should look at Travis. > > > > There was no notice... > > > > ...which is an issue. I added CFLAGS in r67999 but I'm not sure how to do > it properly. Even it it works, I'm not sure if it is correct (it is > probably overriding whatever was set before). The build is fine. Check the value of CFLAGS in include/Make/Platform.make when you don't set it in the environment. I think that autoconf defaults to "-g -O2" for gcc. For a release build, I'd suggest -O2 and -Wall, plus whatever else you want to add. -g will enlarge the binaries with debug information, but "make strip" or "make install-strip" will remove it. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Moving GRASS Python parts to Unicode
Maris Nartiss wrote: > as you might already have noticed, there is a constant stream of > issues containing keywords "encoding" or more often > "UnicodeDecodeError". The main reason behind this is Python 2.x two > types of text strings - byte sequence (one you get with str()) and > Unicode (unicode()). Python 3.x will have only one - Unicode (byte > sequence is not a string any more) thus fixing this frustrating source > of errors. Both versions have both types of string. In 2.x, str() and "plain" string literals create byte strings, while unicode() and u"..." create unicode strings. In 3.x, str() and plain string literals create unicode strings, while bytes() and b"..." create byte strings. The biggest differences between the two are: a) 2.x allows implicit conversions. If you pass a byte string where a unicode string is expected (or vice versa), the string is implicitly converted using the default encoding (which can't be set by a script). 3.x doesn't do this; you get an exception. b) 3.x tries quite hard to maintain the fiction that everything is unicode. E.g. sys.argv contains unicode strings, os.environ uses unicode strings for both keys and values, sys.stdin/stdout/stderr are text streams which return Unicode data. > Moving GRASS Python code to use Unicode internally will make it closer > to Python 3 ready and solve largest part of errors caused by implicit > conversation from encoded text strings to Unicode text strings. I don't particularly care what happens with wxGUI, and using unicode consistently would make sense there, as wx itself uses Unicode. But if you're planning on doing this to grass.script, I'm strongly opposed. It achieves nothing beyond making what should be wxGUI's problem into everyone else's problem. Pretending that everything is unicode only works so long as the rest of the world makes sure not to dispel the illusion. Otherwise, it fails hard. Something as simple as e.g. copying stdin to stdout fails just because the data isn't in the assumed encoding. Bear in mind that the C portion of GRASS (i.e. most of it) doesn't pay any attention to encodings unless it has to. It just passes bytes around. It doesn't care whether the bytes are in any particular encoding, and certainly won't attempt to ensure that data written to stdout or to files is in any particular encoding. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] python: get results of r.stats call into list of lists of numericals instead of list of strings
Moritz Lennert wrote:
> I need some help with python scripting. I'm running r.stats on two
> raster maps such as this:
>
> rstats_results = gscript.read_command('r.stats',
> input_=[a, b],
> flags='n1',
> separator='comma',
> quiet=True)
> results = rstats_results.splitlines()
>
>
> This gives me
>
> ['123,456', '456,789', '987,321']
>
> Does anyone know an efficient way to transform this into a list of lists
> of numbers, such as:
>
> [[123, 456], [456, 789], [987, 321]]
>
> I know I can do this as follows:
>
> mylist = []
> for line in results:
> mylist.append([int(line.split(',')[0]), int(line.split(',')[1])])
>
> But for very large raster maps this does not seem very efficient.
>
> Is there a way of getting the output directly as a list of lists ? Or a
> more efficient transformation ?
For a start, there's no reason to call the .split() method twice:
for line in results:
a, b = line.split(',')
mylist.append([int(a), int(b)])
A list comprehension might be more efficient, e.g.:
mylist = [[int(x)
for x in line.split(',')]
for line in results]
Also, rather than reading the output as a string, I'd use
pipe_command() and iterate over the file:
rstats = gscript.pipe_command('r.stats',
input_=[a, b],
flags='n1',
separator='comma',
quiet=True)
mylist = [[int(x)
for x in line.strip().split(',')]
for line in rstats.stdout]
I don't think that you'll do much better than that in Python. If it's
still too slow, try using numpy.loadtxt() (pre-processing the data
with sed if that function can't read it directly).
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] cp -r vs -R in make
Anna Petrášová wrote: > cp: the -H, -L, and -P options may not be specified with the -r option. > > make: *** [install] Error 1 > > > It seems cp -r (lowercase r) is not portable, see [1]. Mac cp man page says: > > Historic versions of the cp utility had a -r option. This implementation > supports that option; how-ever, however, ever, its use is strongly > discouraged, as it does not correctly copy special files, symbolic links, > or fifo's. > > lowercase r is used on couple of places, mainly include/Make/Install.make. > So my question is if anyone thinks there are some valid reasons for using > lowercase r. "cp" probably shouldn't be used at all. Executable files should be installed with $(INSTALL), non-executable files with $(INSTALL_DATA). In either case, they should only be used to copy a rule's prerequisite to its target (i.e. no unconditional "bulk" copies in the body of an unrelated rule), as in: destination: source $(INSTALL_DATA) $< $@ If the number of files is large, $(wildcard ...) and $(patsubst ...) should be used to construct a list of targets which should then be made prerequisites of the Makefile's "default" rule, with the installation performed by a pattern rule. The only places "cp" is used in the actual GRASS source tree (not including "addons") are in Install.make, the "install" targets of a couple of other *.make files, and the "update-po" target in locale/Makefile. None of these are part of the actual build process. (I just removed the one in simlib; Rules.make already has a pattern rule for installing headers). FWIW, the "portable" solution for recursive copies has historically been to use "tar c ... | tar x ...". -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] eval option in r.mapcalc in a python script
Paulo van Breugel wrote:
> From the r.mapcalc manual, the 'eval' option is described to run more
> complex expressions. But I wonder how I apply the example below in a
> python script?
>
> r.mapcalc << EOF
> eval(elev_200 = elevation - 200, \
> elev_5 = 5 * elevation, \
> elev_p = pow(elev_5, 2))
> elevation_result = (0.5 * elev_200) + 0.8 * elev_p
> EOF
Use grass.script.mapcalc().
The first parameter is a template string, which is passed to
string.Template() (see the Python documentation for the string module)
then instantiated using any additional keyword arguments. This can be
used if you need to be able to substitute parameters into the
expression.
For complex expressions, you can use any of Python's features to
construct the expression. E.g.
assignments = [
"elev_200 = elevation - 200",
"elev_5 = 5 * elevation",
"elev_p = pow(elev_5, 2)"]
result = "elevation_result = (0.5 * elev_200) + 0.8 * elev_p"
expr = "eval(" + ",".join(assignments) + ");" + result
gscript.mapcalc(expr)
This may be useful if you need to be able to dynamically manipulate
the structure of the expression rather than just parameters.
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.mapcalc: warning: field width should have type 'int', but argument has type 'yy_size_t'
Markus Neteler wrote: > >> compiling relbranch70 (likely the same on trunk), I got > > > > Try r67518. > > Issue gone, thanks. > > I spotted another single one: > > lib/db/sqlp/sqlp.tab.c r67559 -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.mapcalc: warning: field width should have type 'int', but argument has type 'yy_size_t'
Markus Neteler wrote: > compiling relbranch70 (likely the same on trunk), I got Try r67518. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Unable to rename cell/null file - NULL file messed up
Paulo van Breugel wrote:
> Can't answer that, but I see that a related (?) ticket
#2434 isn't related to #2712. #2712 relates specifically to the null
file compression changes. #2434 was opened a year before those changes
were started.
> https://trac.osgeo.org/grass/ticket/2434 was retargeted to 7.03? From the
> description that issue seems to be very similar to the problem described
> above?
The problem described above:
> > > WARNING: Unable to rename cell file
> > > 'C:\GIS\GrassJvdS/latlong/YLD_NUT/.tmp/unknown/3988.0' to
> > > 'C:\GIS\GrassJvdS/latlong/YLD_NUT/fcell/Shan3_CA_Renyi_0_0': File exists
> > > WARNING: Unable to rename null file
> > > 'C:\GIS\GrassJvdS/latlong_2005/YLD_NUT/.tmp/unknown/4100.1' to
> > > 'C:\GIS\GrassJvdS/latlong_2005/YLD_NUT/cell_misc/Shan3_CA_Renyi_1_0/null':
> > > File exists
doesn't match the symptoms of either #2434 or #2712. In both of those
cases, the rename() fails with EACCESS ("Permission denied"), which is
symptomatic of the file being open.
This case fails with EEXIST ("File exists"), which suggests that the
destination file wasn't deleted first. On Unix, deleting the original
isn't required (or even desired); rename() will atomically remove any
existing file. Windows generates an error if the destination exists.
So we (try to) remove it first (on all platforms).
remove(path);
if (rename(fcb->temp_name, path)) {
G_warning(_("Unable to rename cell file '%s' to '%s': %s"),
fcb->temp_name, path, strerror(errno));
stat = -1;
}
(close_new() in lib/raster/close.c).
The warning suggests that the file exists and remove() failed to
remove it. On Windows, that can happen if the file is open.
And looking at r.biodiversity.py, I notice:
grass.mapcalc("$outl = -1 * $outl",
outl=out_renyi,
overwrite=True,
quiet=True)
I.e. it's running r.mapcalc with the same map as both input and
output. That isn't guaranteed to work. And in fact it doesn't, because
the close_maps() function (which closes all of the input maps) is
never actually called.
So when it tries to close the output maps, which involves renaming the
new cell/fcell/null files into place, the original cell/fcell/null
files are still open for reading, meaning that the remove() will fail,
as will the subsequent rename().
Having r.mapcalc close the input maps would avoid that, and should be
done, but ultimately that's fixing the symptoms. r.biodiversity
shouldn't be trying to modify out_renyi "in-place". It should create a
new map then, if necessary, replace the original using g.remove and
g.rename.
One problem with in-place modification is that closing an output map
writes a number of support files (history, range, quant, cats,
histogram). But after closing the output maps, r.mapcalc will
sometimes copy some of the metadata (cats, colors, history) from the
input map. Currently, it only does this if the entire expression is
just a map with optional modifiers, but it may get more "intelligent"
in the future. Clearly, that's a problem if the input map has already
been replaced.
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] GRASS 7.1 compiles but won't run
Moritz Lennert wrote: > > However, if you think it would help, I can delete the entire > > installation and checkout again. > > Yes, I have found that from time to time make distclean is not enough. I > haven't dug into this deeply enough to understand why, but there seems > to be some cleaning tasks missing and that corrupts the source tree even > after a make distclean. "svn status --no-ignore" will show everything except for unmodified, version-controlled files. Modified and non-version-controlled files will be shown (even if explicitly ignored due to an svn:ignore property). In a perfectly clean source tree, that command should produce no output. The only difference between "make distclean" and "make clean" is that the former also removes the files generated by "configure" (so you'll get an error if you try to build without first running configure, rather than it using stale files). Here (on Linux), I don't see any ignored files after "make distclean", and the only non-ignored "unknown" files are patches, logs, notes, etc which are there through my own actions (i.e. not as a result of running "make"). It's possible that, on other architectures, the build process may create additional files which should be added to the "clean" rule. Please report any that are found. When updates remove directories, those directories will be removed from SUBDIRS in the parent Makefile, which means that "make clean" won't remove them. "svn update" will delete any removed files, but it won't delete subdirectories which aren't empty (e.g. if "make clean" wasn't performed first, the OBJ.$(ARCH) directory will remain and thus the parent directory won't be deleted by "svn update"). Similarly, if files or directories which are marked for cleaning via $(EXTRA_CLEAN_FILES), $(EXTRA_CLEAN_DIRS) or $(CLEAN_SUBDIRS) are removed, they'll typically also be removed from those variables in the relevant Makefile, and thus won't be removed by "make clean". They typically won't be explicitly listed as build targets or dependencies either, but their mere presence may still affect compilation. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Broken: GRASS-GIS/grass-ci#478 (master - e2f3140)
Vaclav Petras wrote: > I'm not getting any errors after cd into the directories reported by make. > I don't know how to fix this. The header is installed when making lib/db/dbmi_driver, but lib/db/dbstubs is typically made first. It will probably work for incremental builds (which is probably why I failed to notice this before committing) and may work for parallel builds. r66817 moves dbstubs.h to lib/db/dbmi_base, which is an explicit pre-requisite of both dbmi_driver and stubs. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Return value from g.copy is one when --overwrite - bug or feature?
Markus Neteler wrote: > It should probably not silently fail. > > The code in question is in > lib/manage/do_copy.c M_do_copy() returns a status. g.copy's main() checks that and uses it to set the exit status, but it doesn't generate an error or warning if the status is non-zero. If G_recursive_copy() or M_do_copy() raised an error, that would result in the program aborting on the first error; at present, it copies as much as it can. Realistically, g.copy() should be generating the error at the bottom of main() if result is non-zero. E.g. -exit(result); +if (result) +G_fatal_error(...); +return 0; > and potentially there to be added strerror(errno) as for example in > lib/gis/mapset_msc.c That would require G_recursive_copy() to call G_warning() (with the appropriate strerror(errno)) for each failed system call. Either that, or G_recursive_copy() would have to generate a "log" of failures so that the caller can report them. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [BUG] or at least unclear error message
Rainer M Krug wrote: > I have a mega location with 532 read-only mapsets with each around 2500 > mapsets > (plus PERMANENT and an read-write analysis mapset). > > When using g.list type=rast mapset=* I get the following error: "Unable > to open directory". I guess this is caused by to many open files as I > can execute the command g.list type=rast mapset=... for this mapset > without problems. > > (transcript below) > > Is this the case? Could the error message be clarified (I fear not) or a > different approach be used to list the maps? G_ls2() doesn't call closedir() when it has finished reading the directory. Fixed in trunk with r66719. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)
Markus Neteler wrote: > > At a minimum, I may change the thread pool code (lib/gis/worker.c) so > > that the default number of workers (if WORKERS isn't set) is zero, > > i.e. multiple threads will only be used if specifically requested by > > the user. > > Maybe better since this question came up a few times. Done in r66731. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Open/close db drivers order
Radim Blazek wrote: > Should it better go to the original program between fork() and execve() > or into the driver executable? Ideally, it should be done by whatever creates the descriptor in the first place. Doing it when spawning a DBMI driver (whether in the parent or in the child) only avoids the specific case where the process holding onto the descriptor is a DBMI driver. There are other types of child process which might be spawned (possibly invisibly to GRASS, e.g. an external library spawning a "helper" process), and the consequences are always the same. If I was going to add a mechanism for closing all descriptors other than 0/1/2, it would probably be as an option to G_spawn_ex(). Also, the situation is potentially worse on Windows, as handles can't be enumerated (and even if they could, process creation is atomic; there's no way to execute code in the child process prior to the start of the new program). If CreateProcess() is called with bInheritHandles set to TRUE (which is required for stdin/stdout/stderr to be inherited), all inheritable handles are inherited. > > The only robust solution is for the original program to systematically > > set the close-on-exec flag on every descriptor *except* those which > > are meant to be inherited. > > The original program may be using third party libraries which will > never care of. For libraries, it's even more important that they set the close-on-exec flag on "internal" descriptors, because nothing else can. Ultimately, you can have one library which creates "internal" descriptors and another library which spawns "internal" helper processes, and there's not much that the main program can do about the interaction between these. It's generally considered the responsibility of the code creating descriptors to set the close-on-exec flag, because that's easier (and safer) than expecting code which spawns child processes to close descriptors which it knows nothing about. In this case, the key issue is that the DBMI client library needs to set the close-on-exec flag on p1[WRITE], as it's important that this descriptor doesn't get duplicated. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Open/close db drivers order
Radim Blazek wrote: > But there may be other file descriptors not related to db drivers. Is > it possible to close all of them in driver process? Maybe. When fork() clones a process, the descriptor table is cloned along with everything else. When the process changes its program with execve(), any descriptors without the FD_CLOEXEC flag remain open. But the new program won't know of their existence, where they came from, what they refer to, etc. The "knowledge" about the descriptors only exists in the original program, so closing specific descriptors (or setting their close-on-exec flags) has to be done prior to the execve(). Once the program is changed and that information is lost, about the only option available to the driver is to close *every* open descriptor except for 0, 1 and 2 (standard input/output/error). This needs to be done before calling any library function which might create additional descriptors (which in practice means almost any library function). Even that is complicated by the fact that there's no standard mechanism for discovering the highest open descriptor number. You just have to iterate until reaching some arbitrary maximum, or until some (again, arbitrary) number of consecutive descriptors are found to be unused (by close() failing with EBADF), and hope that's all of them. The only robust solution is for the original program to systematically set the close-on-exec flag on every descriptor *except* those which are meant to be inherited. The fact that descriptors are always inherited across fork() and preserved across execve() by default is generally considered to be a mistake in the POSIX API, but we're now stuck with it. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Open/close db drivers order
Radim Blazek wrote:
> I found, that db drivers must be closed in reverse order to the order
> in which were opened, otherwise it hangs (Linux). I vaguely remember
> that this was a problem years ago. Is it still an issue, anybody knows
> more about the problem?
It sounds like an issue with inheriting file descriptors. I.e.:
1. Opening the first driver causes the main process to get some file
descriptors for the pipes connecting it to the driver
2. Opening the second driver will result in the driver inheriting
copies of those descriptors from the main process.
3. When the main process closes the write end of the pipe whose read
end is the first driver's stdin, the first driver won't get EOF on its
stdin because the second driver still has a copy of the descriptor for
the write end of the pipe and hasn't closed it (it doesn't even know
it has it).
If that's what's happening, setting the close-on-exec flag for each
descriptor should fix it. E.g.
void close_on_exec(int fd)
{
#ifndef __MINGW32__
int flags = fcntl(fd, F_GETFD);
fcntl(fd, F_SETFD, flags | FD_CLOEXEC);
#endif
}
close_on_exec(p1[READ]);
close_on_exec(p1[WRITE]);
close_on_exec(p2[READ]);
close_on_exec(p2[WRITE]);
p1[WRITE] is the one which is most likely to matter; that's the one
which needs to be closed (in all processes) for the child to get EOF
on stdin.
p1[READ] and p2[WRITE] are closed in the parent after the child has
been spawned, so they won't be inherited by subsequent child
processes.
--
Glynn Clements <gl...@gclements.plus.com>
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)
Dylan Beaudette wrote: > > If it's a pthread issue, using "export WORKERS=0" from the shell > > before running t.rast.mapcalc should make it go away. > > I don't think that it has anything to do with pthreads... Does the > "--without-pthread" configure argument actually work? --without-pthread is the default. You can check whether GRASS was built with pthread support by searching for HAVE_PTHREAD_H in include/grass/config.h. The configure script only checks for (and defines that macro) if --with-pthread is given. > > r.mapcalc uses a mutex for each map to prevent multiple threads > > accessing a single map concurrently. Accessing different maps from > > multiple threads should be safe ... for normal GRASS rasters, and > > provided that no MASK raster is present (otherwise it ends up > > accessing the MASK raster from multiple threads concurrently, which > > results in the "Error reading raster data ..." error, due to a race > > condition between the lseek() and the read()). > > Got it. This is valuable information. Is it documented anywhere other > than GRASS-user and GRASS-dev mailing lists? How hard would it be to > issue a warning when someone tries parallel access to the MASK? It > would seem that some of the new t.* modules could be affected by this > reality as many of them support parallel processes. Is it the duty of > the module author to warn of such pitfalls? The MASK issue should have been fixed with r65591 (in trunk, not sure if it has been backported to releasebranch_7_0 yet). Ultimately, if a module uses pthreads itself, it has to ensure that any functions which aren't thread-safe aren't called concurrently from multiple threads. Accessing different maps from multiple threads is "intended" to be safe. But the issue with the MASK was overlooked, as was the potential for issues with GDAL-linked rasters. I'm starting to think that r.mapcalc's pthread support to may be more trouble than it's worth. Unless the expression is particularly complex, r.mapcalc tends to be I/O-bound, and pthreads doesn't really help with that. At a minimum, I may change the thread pool code (lib/gis/worker.c) so that the default number of workers (if WORKERS isn't set) is zero, i.e. multiple threads will only be used if specifically requested by the user. > > But for GDAL-linked (r.external) rasters, there may be re-entrancy > > issues with GDAL itself, meaning that even accessing different maps > > from multiple threads is unsafe. > > This is troubling. Is there anyway to empirically determine what is > safe? Last week worked on a project where I was processing daily > rasters for a 30 year interval, involving 3 climatic variables. Due to > the large number of files, I had to work with "external" data sources. > For the most part, everything went as expected when the results of > (parallel) r.mapcalc expressions were CELL maps. Not so when the > results were FCELL or DCELL maps. The main problem with GDAL is that it's essentially a collection of distinct modules in a common framework. Unless they have (and enforce) a policy that GDALRasterIO() must be re-entrant (at least for different maps), it may be the case that concurrent access is safe for some formats but not others. > > Fixing that is potentially problematic. r.mapcalc doesn't know whether > > its inputs are GDAL-linked (this feature is supposed to be transparent > > to modules; there isn't a Rast_is_gdal(int fd) or similar function), > > and lib/raster itself doesn't have any knowledge of pthreads (lib/gis > > provides functions for managing a thread pool; r.mapcalc/r3.mapcalc > > are currently the only modules using these functions, and r3.mapcalc > > specifically sets WORKERS=0 because lib/raster3d isn't thread-safe). > > OK. Does setting "WORKERS=0" do anything when GRASS is compiled > without pthreads? No. If you're experiencing these problems with WORKERS=0 or with GRASS compiled without pthread support, then the problem is something else entirely. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)
Dylan Beaudette wrote: > I can now verify that t.rast.mapcalc is creating some raster maps with > corrupt (?) data. Corrupt in the sense that subsequent reading of the > maps results in the "Error reading raster data for row ..." error. > Just in case anyone is interested, I have opened a ticket for more > informative errors raised by lib/raster/get_row.c > (https://trac.osgeo.org/grass/ticket/2762). > > As previously reported, errors seem to occur about 50-60% of the time > and _do not_ appear to be related to the number of concurrent > t.rast.mapcalc instances. t.rast.mapcalc relies upon r.mapcalc. If it's a pthread issue, using "export WORKERS=0" from the shell before running t.rast.mapcalc should make it go away. r.mapcalc uses a mutex for each map to prevent multiple threads accessing a single map concurrently. Accessing different maps from multiple threads should be safe ... for normal GRASS rasters, and provided that no MASK raster is present (otherwise it ends up accessing the MASK raster from multiple threads concurrently, which results in the "Error reading raster data ..." error, due to a race condition between the lseek() and the read()). But for GDAL-linked (r.external) rasters, there may be re-entrancy issues with GDAL itself, meaning that even accessing different maps from multiple threads is unsafe. Fixing that is potentially problematic. r.mapcalc doesn't know whether its inputs are GDAL-linked (this feature is supposed to be transparent to modules; there isn't a Rast_is_gdal(int fd) or similar function), and lib/raster itself doesn't have any knowledge of pthreads (lib/gis provides functions for managing a thread pool; r.mapcalc/r3.mapcalc are currently the only modules using these functions, and r3.mapcalc specifically sets WORKERS=0 because lib/raster3d isn't thread-safe). -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] installing v.mapcalc fails
Markus Neteler wrote: > <p.vanbreu...@gmail.com> wrote: > ... > > Sorry for the late follow up. The add-on installs now. When trying to run > > it, however, nothing happens. > > It does not have a parser (yet?), so I suppose it reads from stdin. Yes. But it doesn't have any code to read or write GRASS vectors, either. v.mapcalc appears to be a work-in-progress which hasn't actually made noticeable progress since it was added. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [GRASS-SVN] r66413 - grass-addons/grass7/vector/v.kriging
Vaclav Petras wrote: > I like the idea of a separate header but I would leave the implementation > in lib/gmath. Or is this going too much against how other libraries are > done? I'm also not sure how the prefixes should work, but these seems OK > since there are functions with same prefix (G_) in several libraries. Having separate headers for self-contained sections of a library isn't an issue. Ideally, if a localised section of a library has dependencies not required for the rest of the library, that section is a candidate for being a separate library (so that a lack of the dependency only disables the features which require it, rather than the entire library). But this isn't rigorously followed (e.g. libgis has compile-time dependencies on iconv, pthread, regex, and zlib, even though each of those libraries is only used for isolated features). > Having la.h as a separate header would conveniently hide the compilation > issue on that Debian server. > > So, should I remove la.h from gmath.h? I think so. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] GRASS will not not find laslib
Markus Neteler wrote: > > But it has been obsoleted by wxpyimgview (a wxPython version), so I > > see no reason not to simply remove it (and the wxWidgets configure > > checks as well). > > Sounds reasonable for trunk. Done in r66476 (wximgview) and r66477 (wxWidgets configure checks). The latter may have an impact upon macosx/app/Makefile, which references MACOSX_ARCHS_WXPYTHON (which no longer exists). -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] GRASS will not not find laslib
Michael Barton wrote: > But wximgview would not compile with what appears to be a 32/64 bit conflict wximgview is (AFAIK) the only program which links against the wxWidgets libraries. But it has been obsoleted by wxpyimgview (a wxPython version), so I see no reason not to simply remove it (and the wxWidgets configure checks as well). -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [GRASS-SVN] r66413 - grass-addons/grass7/vector/v.kriging
Vaclav Petras wrote: > I was not able to determine from svn why gmath.h contains la.h. AFAICT, it's because the functions are in lib/gmath. Of course, that then brings up the question why they're there, rather than in a separate library ... Incidentally, the whole of lib/gmath/la.c is conditionalised upon #if defined(HAVE_LIBLAPACK) && defined(HAVE_LIBBLAS) but individual functions (and even sections of functions) are conditionalised as necessary. And there are a few functions for which we could reasonably supply our own implementation if BLAS/LAPACK aren't available. E.g. G_matrix_product() and G_vector_norm_euclid() should be straightforward. AFAICT, G_matrix_LU_solve() is the only function that's moderately complex. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [GRASS-SVN] r66413 - grass-addons/grass7/vector/v.kriging
Markus Neteler wrote: > >> Sorry for being -pedantic but I don't think we should leave the code > >> messy. > > > > +1 > > Then it needs to be removed everywhere here: > > cd grass-addons > find . -type f | grep -v pristin | xargs grep 'la.h>' > ./grass6/imagery/i.spec.unmix/global.h:#include > ./grass6/imagery/i.spec.unmix/la_extra.c:#include > ./grass6/imagery/i.spec.grass63/include/gmath.h:#include > ./grass6/imagery/i.spec.grass63/lib/gmath/la.c:#include > ./grass7/imagery/i.spec.unmix/global.h:#include > ./grass7/vector/v.kriging/local_proto.h:#include Sorry for joining in late, but from my understanding of the code, it seems that the inclusion of should be in the modules, not in . defines a number of macros plus the G_math_spvector type. It then includes which contains a number of function declarations, some of which use G_math_spvector in their return or parameter types. also includes , but doesn't depend upon anything defined in that header. If the include of is removed, compiling a source file containing only #include succeeds. Consequently, there is no need for to include . Having one header for "base" functionality which is always available and another for additional functionality which is only available when GRASS was built with BLAS/LAPACK support seems cleaner than having a single header expose both, particularly when the two have already been separated. Additionally, requiring individual modules to include explicitly would make it clear when code depends upon BLAS/LAPACK. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] v.mapcalc missing UI
Martin Landa wrote: > does anyone know how v.mapcalc should work? It seems to me that the > module lacks an UI, no options, no G_parser(). It leads to the fact > that > > v.mapcalc --help > > just freezes. v.mapcalc reads the expression from stdin (see yylex.c); argv is ignored. Not only does it not call G_parser(), it doesn't call G_gisinit() either. The only GRASS functions which it calls are: G_calloc G_malloc G_realloc G_fatal_error G_free G_gettext G_message G_warning I suspect that it's probably a long way from being functional at present. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] installing v.mapcalc fails
Moritz Lennert wrote: > > yylex.c:9:27: fatal error: v.mapcalc.tab.h: No such file or directory > > > >#include "v.mapcalc.tab.h" > > > > ^ > > > > compilation terminated. > > > > make: *** [OBJ.x86_64-unknown-linux-gnu/yylex.o] Error 1 > > > > ERROR: Compilation failed, sorry. Please check above error messages. > > Glynn, can you help us on this ? We need to find the correct Makefile > for the v.mapcalc addon [1] That header is generated by yacc (or byacc, bison, etc) via a pattern rule in include/Make/Compile.make: %.output %.tab.h %.tab.c: %.y $(YACC) -b$* $(YFLAGS) $< The problem with v.mapcalc is that it tries to compile yylex.c before v.mapcalc.tab.h has been generated, which fails. It requires: -yylex.c: v.mapcalc.tab.h +$(OBJDIR)/yylex.o: v.mapcalc.tab.h The header is a dependency of the object file; the source file doesn't have dependencies (it's a source file, not generated). Committed in r66392. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] How to get data from a location where the user does not own any mapset?
Radim Blazek wrote: > >> Perfect, that is what I need. Why GRASS_SKIP_MAPSET_OWNER_CHECK should > >> not be set by programs for read only access? > > > > If you only need to read a mapset, you shouldn't need to make it the > > current mapset, nor should you try. > > The mapset I need to read from does not have to be current, but one > mapset must be current and G_gisinit() checks if user is owner of > current mapset. If the user is not owner of any mapset in the > location, what can be set as current mapset? Use a mapset which the user does own, or create one, and use G_switch_env() to select a different location and mapset temporarily. Be sure to switch it back before performing any operation which may write to the "current" mapset (modifying WIND/VAR, creating maps, or creating temporary files). -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] How to get data from a location where the user does not own any mapset?
Radim Blazek wrote: > Perfect, that is what I need. Why GRASS_SKIP_MAPSET_OWNER_CHECK should > not be set by programs for read only access? If you only need to read a mapset, you shouldn't need to make it the current mapset, nor should you try. Making a mapset the current mapset means that any change to e.g. the WIND or VAR files will modify the files in that mapset. Creating temporary files will create them in that mapset. And so on. Also, using g.mapset to make a mapset the current mapset will try to write the .gislock file there, and will fail if that file exists and the creator is still alive. Needless to say, any operation which attempts to write to the mapset will fail if the filesystem permissions don't allow the operation. An environment variable won't change that. And if a session is using a mapset as its current mapset, the contents of the WIND and VAR files in that mapset can change at any time. > > In short, it's a safety mechanism against people trying to share > > directories by making them group-writeable without fully understanding > > the implications. > > I am only interested in read only access. The purpose is to be able to > read and display data in QGIS. I believe that read only access to a > location where the user does not own any mapset is a legal > requirement, isn't it? Many GRASS library functions assume the existence of a current mapset which can be modified. The few modules which access a location other than the one containing the current mapset use G_create_alt_env() and G_switch_env() to temporarily select the "other" location while performing specific (read-only) operations, then switch back afterwards. This is something of a kludge, and probably somewhat error-prone, but it's all we have. There have been some discussions on cleaning this up, but they never really got anywhere. For now, even accessing data in other mapsets in the same location relies on a certain amount of luck. Changes to the cell/fcell files are atomic; changes to other files aren't (at least, not by design; changes to cellhd files end up being atomic purely by virtue of the fact that cellhd files are typically smaller than the stdio buffer). -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] How to get data from a location where the user does not own any mapset?
Radim Blazek wrote: > Is it possible to access data through GRASS modules in a location > where the user does not own any mapset? Each GRASS module calls first > G_gisinit() which checks if current mapset returned by G_mapset() is > owned by current user. If it is not, it ends with fatal error. How to > get around this for locations where the user does not own any mapset? In general, you can only access data in the current location. For most modules, it's meaningless to access data from a different location as the projection will be wrong. A handful of modules (e.g. r.proj, v.proj) support accessing data from a different location, and in such cases there's no need for the user to own the source mapset, or even to own any mapset in the source location. Since r54264, it is possible to bypass the ownership check by setting GRASS_SKIP_MAPSET_OWNER_CHECK to any non-empty string. This was added mostly for accessing VFAT/NTFS-format external drives or network shares where the ownership checks don't behave correctly. It should not be used on multi-user systems, nor set by programs.. But other than that, you cannot make a mapset the current mapset (i.e. the one you can modify) unless you own it. You can access data from any mapset for which you have the relevant filesystem permissions (i.e. read permission for files, execute permission for the ancestor directories, read permission for any directories which need to be enumerated). Originally this restriction was related to locking (the lock file was created in the user's home directory, so being able to set the current mapset to a directory which you didn't own meant that you could have multiple sessions modifying a mapset concurrently). That's no longer an issue (the lock file is created in the mapset directory). Howver, the other issue is that allowing someone other than the owner to modify the directory can result in problems for its owner. Creating a new map typically creates a subdirectory of cell_misc for that map, and some files within it (e.g. a "null" file). Any new files and subdirectories will be owned by their creator (not the mapset's owner) and will have permissions determined by the current umask. The result is that there's no guarantee that tha mapset's owner will be able to delete (or even access) such files or directories. In short, it's a safety mechanism against people trying to share directories by making them group-writeable without fully understanding the implications. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Display library documentation?
Moritz Lennert wrote: > Glynn, as you are the one who significantly rewrote the display library, > do you have any documentation or notes stored somewhere ? No. And I'm not familiar with the parts related to symbols; I haven't touched that (beyond keeping it in sync with other changes to the display library). So far as coordinates go, for drawing geographic data, the usual procedure is to call D_setup() and use cartographic coordinates directly. For other data (e.g. legends), D_setup2() sets a user-defined coordinate system while D_setup_unity() allows screen coordinates to be used directly. The library deals with 3 coordinate systems: U (user) coordinates are used by most D_* functions which take coordinates (the exceptions being the functions which convert coordinates from other coordinate systems). A (array) coordinates correspond to the region grid, and are used implicitly for drawing raster data. D (display) coordinates are screen pixel coordinates. The display library converts U and A coordinates to D coordinates, which are passed to driver functions (COM_*). The low-level functions for setting coordinate systems and performing conversions are in cnversions.c. Higher-level convenience functions for setting coordinate systems are in setup.c. D_setup() sets the U coordinate system to cartographic coordinates so that the current region fits exactly inside the current display frame while maintaining aspect the ratio. D_setup2() sets the U coordinate system to a user-defined coordinate system so that the specified rectangle fits exactly inside the current display frame while optionally maintaining the aspect ratio. D_setup_unity() sets the U coordinate system to match the D coordinate system. This exists mainly to simplify the transition for existing code. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Vect_rewrite_line using off_t as line id
Radim Blazek wrote: > Can I get from GRASS includes the real type of off_t which was used > when GRASS was compiled? Not easily. Or reliably. You could use the compiler switches from Platform.make and include in the hope that it's enough to get the same off_t that GRASS used. Ultimately, the only place where the actual off_t size is "known" is in compiled code. It might be worth adding this information to the output of g.version. -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.resample.interp also does not behave according to manual
Michael Barton wrote: > g.region raster=ziqlab_region_resamp30m > r.resamp.interp --overwrite input=Mediterranean_east_SRTM90m@PERMANENT > output=ziqlab_region_resamp30m_test method=lanczos > > ERROR: Reading raster map <Mediterranean_east_SRTM90m@PERMANENT> > request for row 1216 is outside region 1. Does it happen with a method other than lanczos? 2. Does the following change fix it? --- raster/r.resamp.interp/main.c (revision 66085) +++ raster/r.resamp.interp/main.c (working copy) @@ -111,11 +111,11 @@ double north = Rast_row_to_northing(0.5, _w); double south = Rast_row_to_northing(dst_w.rows - 0.5, _w); int r0 = (int)floor(Rast_northing_to_row(north, _w) - 0.5) - 2; - int r1 = (int)floor(Rast_northing_to_row(south, _w) - 0.5) + 3; + int r1 = (int)floor(Rast_northing_to_row(south, _w) - 0.5) + 4; double west = Rast_col_to_easting(0.5, _w); double east = Rast_col_to_easting(dst_w.cols - 0.5, _w); int c0 = (int)floor(Rast_easting_to_col(west, _w) - 0.5) - 2; - int c1 = (int)floor(Rast_easting_to_col(east, _w) - 0.5) + 3; + int c1 = (int)floor(Rast_easting_to_col(east, _w) - 0.5) + 4; src_w.south -= src_w.ns_res * (r1 - src_w.rows); src_w.north += src_w.ns_res * (-r0); -- Glynn Clements <gl...@gclements.plus.com> ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] g.region error msg with empty WIND file
Markus Neteler wrote: ERROR: Field projection missing ... shows after longer debugging that really the WIND file was simply 0 bytes: ls -la /grassdata/eu_laea/sp_modis_swath/WIND -rw-r--r-- 1 sajid gis 0 Aug 6 11:55 /grassdata/eu_laea/sp_modis_swath/WIND How to change lib/gis/rd_cellhd.c to detect this and write out an informative message? The higher up the call chain you detect it, the more specific the error message can be, but the fewer cases it will handle. The lowest point is immediately prior to the test which generates the existing error, e.g.: --- lib/gis/rd_cellhd.c (revision 65851) +++ lib/gis/rd_cellhd.c (working copy) @@ -165,6 +165,8 @@ continue; } } +if (line == 1) +G_fatal_error(_(Region definition is empty)); if (!TEST(F_PROJ)) G_fatal_error(_(Field %s missing), projection); if (!TEST(F_ZONE)) That handles the same set of cases as the existing error, but isn't significantly more informative; it's just clarifying that *all* of the fields are missing rather than just the projection. Catching it in G__read_Cell_head() will allow you to diagnose that it's a file that is empty: --- lib/gis/rd_cellhd.c (revision 65851) +++ lib/gis/rd_cellhd.c (working copy) @@ -77,6 +77,9 @@ count++; } +if (!count) +G_fatal_error(_(Region file is empty)); + G__read_Cell_head_array(array, cellhd, is_cellhd); count = 0; But this won't catch the case where GRASS_REGION is being used, and it won't allow you to report *which* file is empty. Catching it in G_get_element_window() would allow you to report the filename, but additionally won't catch calls from Rast_get_cellhd() to read the mapset/cellhd/map file. --- lib/gis/get_window.c(revision 65851) +++ lib/gis/get_window.c(working copy) @@ -118,6 +118,10 @@ G_fatal_error(_(Unable to open element file %s for %s@%s), element, name, mapset); +G_fseek(fp, 0, SEEK_END); +if (!G_ftell(fp)) +G_fatal_error(_(Region file %s/%s/%s is empty), mapset, element, name); +G_fseek(fp, 0, SEEK_SET); G__read_Cell_head(fp, window, 0); fclose(fp); } If you want to handle more of the cases while reporting the filename, G__read_Cell_head() and G__read_Cell_head_array() could be made to take the filename as an extra parameter so that it can be included in error messages. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Add support for Python3
Vaclav Petras wrote: However, now I see at least 3 tests which were not failing before [3, 4, 5]. It seems like a trivial issue caused by some automatic changes (percent operator is missing). This issue should be visible on any machine. The bug which is causing the test to fail has been present in v.db.addtable for 2 years (since r56701). If the tests only started failing recently, it can only be because of changes to how v.db.addtable was invoked. A database connection having a schema option where none existed before would seem likely. The error arises from: if schema is not '': table = '%s.%s' (schema, table) where schema is set by: kv = grass.db_connection() database = kv['database'] driver = kv['driver'] schema = kv['schema'] -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Add support for Python3
Vaclav Petras wrote: table = '%s.%s' (schema, table) TypeError: 'str' object is not callable As I said, I think it is missing percent operator. Probably trivial to fix but we should understand how it happen to be in the source code. It's a bug which was introduced in r56701: https://trac.osgeo.org/grass/changeset/56701/grass/trunk/scripts/v.db.addtable/v.db.addtable.py It's unrelated to the Python3 conversion. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Error when compiling using homebrew - says liblas but is gdal???
Rainer M Krug wrote: | configure:6181: checking for liblas-config | configure:6238: clang -o conftest -g -O2 -I/usr/local/opt/gettext/include -I/usr/local/Cellar/liblas/1.8.0/include -I/usr/local/include -I/usr/local/include -I/usr/local/include conftest.c -L/usr/local/Cellar/liblas/1.8.0/lib -llas -llas_c -L/usr/local/lib /usr/local/lib/libboost_program_options-mt.dylib /usr/local/lib/libboost_thread-mt.dylib /usr/local/Cellar/gdal/1.11.1_2/lib/libgdal.dylib /usr/local/lib/libgeotiff.dylib /usr/local/lib/libtiff.dylib 15 | clang: error: no such file or directory: '/usr/local/Cellar/gdal/1.11.1_2/lib/libgdal.dylib' The configure script runs liblas-config --libs to obtain the libraries. liblas-config is saying that it needs /usr/local/Cellar/gdal/1.11.1_2/lib/libgdal.dylib but the linker complains that the file doesn't exist. It may be that libLAS was built against a GDAL library which has since been removed. In which case, you need to either restore that library or re-build libLAS to use and existing version of GDAL (or without GDAL; the dependency is optional). -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [gdal-dev] Setting NODATA to -nan
Even Rouault wrote: - I've had a look at https://trac.osgeo.org/grass/changeset/65602 and my understanding is that now a pixel value at NaN will be considered as null, even if null_val is not NaN (but this is perhaps intended, if so, ignore this). - This is just for the sake of my curiosity: do you know how -nan pixel values can be generated by GRASS in practice ? (This was what triggered all this: the GDAL reporter was surprised to see pixels at -nan and nodata at nan, and thought that something was wrong) Rast_set_d_null_value() and Rast_set_f_null_value() use the all-ones bit pattern. This is one of the many NaN values (anything with an all-ones exponent and a non-zero mantissa is NaN). As the topmost bit (i.e. the sign bit) is set, it's possible that some code would consider that to be -NaN. E.g. code which writes a leading - based upon the sign bit before considering the other components would do so. Rast_is_d_null_value() and Rast_is_f_null_value() treat any NaN as null (specifically, they test whether the value is unequal to itself). At one time, these functions (or rather, their predecessors) checked explicitly for the all-ones pattern, but this was changed (in r33717 and r33752) to improve robustness. Apart from code explicitly setting a value to null, NaNs can arise from calculations (0.0/0.0, sqrt(x) or log(x) for x0, asin(x) or acos(x) for abs(x)1, etc), and there's no guarantee as to exactly which NaN representation will result. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] chrcat
Martin Landa wrote: recently I needed to append char to string [1], I locally added a new static function to file_name.c. This function will be needed on other places too, would make sense to added it to GRASS API (string.c) as G_chrcat(), what do you think? If it's needed, lib/gis/string.c is the logical place for it to go. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] what is the meaning of: Error reading raster data for row 239 of MASK
Moritz Lennert wrote: Ok. In any case, this probably a candidate for backporting to grass70release before the upcoming release. Can it be backported as such ? Yes. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] what is the meaning of: Error reading raster data for row 239 of MASK
Moritz Lennert wrote: One thing I noticed is that on the one test case I used here for testing your fix, running with WORKERS=0 is slightly faster than without setting it. I didn't test rigorously, but is that expected ? Maybe. It avoids the overhead of switching threads. And using multiple threads only provides a gain if it results in using cores which would otherwise be idle. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Fwd: Re: [gdal-dev] Setting NODATA to -nan
Yann Chemin wrote: it might be that some communication improvement from GRASS-GDAL could be done? is there a clear NODATA/NAN definition understood within GDAL that we could use within r.out.gdal as a target NODATA value whenever anything than a number is used (i.e. NaN, nan, -nan, helloworld, mynameisbee, etc.) typically MODIS users are used to -28768 and similar standard NODATA numbers, but that maybe another story. IMHO, anything using NaN as a no-data value should treat all NaNs equally. The reason being that the only way to reliably propagate the representation for a NaN is to memcpy() it. Assignment only preserves NaN-ness, not necessarily the representation. Actually, r.external is doing it wrong, as it expects any NaNs supplied by GDAL to exactly match the representation generated by Rast_set_d_null_value() (the all-ones bit pattern). Fixed in r65602. And if the no-data value isn't NaN, you have the issue of comparing floating-point values for equality. Well, that will probably work so long as the chosen no-data value is an integer. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] what is the meaning of: Error reading raster data for row 239 of MASK
Moritz Lennert wrote:
I don't know how to debug this...
Can you identify a repeatable test case?
If I could make it happen, I could debug it.
You can get a location names TEST here:
http://tomahawk.ulb.ac.be/moritz/mask_bug_testlocation.tgz
This contains only a PERMANENT mapset.
In that mapset, launch the following command:
r.mask vect=hull; for map in $(g.list rast pat=firm_rate*); do echo
$map ; r.mapcalc temp_prob = float($map) / sum_rates --o --q; done;
r.mask -r
I get the error arbitrarily for different firm_rate_* maps, sometimes
only for one, sometimes for many, but at each run its for different
maps.
So it's non-deterministic (I'm getting one error for every 10-20
passes over the data, i.e. every 1200-2500 commands), and only applies
to r.mapcalc.
My first guess was a race condition related to pthreads. I tried
export WORKERS=0
before running the test, and it hasn't happened since.
And actually I'm now fairly certain as to the specific cause.
When compiled with pthread support, r.mapcalc has a mutex for each map
to prevent concurrent access to a single map from multiple threads.
Concurrent access to different maps (and to core lib/gis and and
lib/raster functionality) from different threads is supposed to be
safe (see r34485 and the interval surrounding it), but the MASK was
overlooked.
If a MASK is in use, reading a row from any raster map will read the
corresponding row from the MASK, and there's nothing to prevent
different threads from concurrently accessing two different maps and
thus accessing the MASK.
So, in read_data_{compressed,uncompressed,read_data_fp_compressed} in
lib/raster/get_row.c we have code like:
if (lseek(fcb-data_fd, (off_t) row * bufsize, SEEK_SET) == -1)
G_fatal_error(_(Error reading raster data for row %d of %s),
row, fcb-name);
if (read(fcb-data_fd, data_buf, bufsize) != bufsize)
G_fatal_error(_(Error reading raster data for row %d of %s),
row, fcb-name);
If multiple threads execute this code concurrently, you can end up
with the calls being interleaved like so:
Thread 1Thread 2
lseek
lseek
read
read
meaning that the file offset has changed betwee the lseek() and the
read() (this is why X/Open and POSIX added pread(), but that's still
relatively new).
This only results in an error at the end of the file (the first read()
will leave the file offset at EOF, so the second read() fails), but in
other situations it's likely causing the wrong row of the MASK to be
read.
A possible quick fix:
if (R__.auto_mask 0)
putenv(WORKERS=0);
A slightly better fix would be to check for masking and if it's
enabled, have a single mutex which guards *all* raster reads so that
even concurrent access to different maps is blocked. Unlike the above
hack, this still allows computations to be executed in parallel.
Better still would be to guard access to the MASK so that the other
aspects of raster input can be parallelised (raster I/O is still a
major bottleneck, and mostly because of processing rather than actual
disc access).
But that would involve either adding pthread code directly into the
base raster input code in lib/raster/get_row.c (undesirable) or at
least adding a mechanism to allow r.mapcalc to hook into it to provide
the mutex.
--
Glynn Clements gl...@gclements.plus.com
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] what is the meaning of: Error reading raster data for row 239 of MASK
Moritz Lennert wrote: So, for me, the best solution at this stage is to just set WORKERS to 0 ? That should work. Also, the issue should be fixed by r65591. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] what is the meaning of: Error reading raster data for row 239 of MASK
Moritz Lennert wrote: I don't know how to debug this... Can you identify a repeatable test case? If I could make it happen, I could debug it. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Parser checking output maps but not input maps
Vaclav Petras wrote: For scripts which use g.parser for argument parsing, if G_parser() fails g.parser itself fails, which in turn causes the script to fail. Python scripts should normally terminate on an exception if a spawned command fails, whereas shell scripts normally ignore the status of any spanwed commands. For Python this is ensured by checking @ARGS_PARSED@ in g.parser output in grass.script.core.parse() [1]. For Bash I have no idea how it is/was done (e.g. in version 6). bash scripts exec g.parser in the current process, in place of bash. g.parser then re-executes the calling script with the option values stored in environment variables. If g.parser fails, the script never gets re-executed, and the exit status of g.parser is the exit status of the process which was originally running the script. However, my problem is that in C if you have an input map, you don't have to check that whether it exists. While in Python you have to check if it exists otherwise the first module (subprocess) you call with it will unexpectedly fail. I hit this issue when I was trying to write documentation for writing scripts and I wanted to do it the right way (as opposed to hopping that some random script I pick in the source code follows well the undocumented API). The question now is if we want to fix the inconsistency in between writing C and Python (and error reporting with overwrite). Or if we say that the current state is good enough as long as it is documented and there is some convenient function to check existence of a map. I don't think I exhausted all options, so if somebody has an idea or is willing to implement something, that would be great. If you consider GRASS modules as being the Python equivalent of GRASS API functions in C, then there isn't an inconsistency. Rather than explicitly checking for a map, we rely upon whatever tries to use generating a fatal error if it doesn't exist. If you want to directly check for the existence of a map, use grass.script.find_file() (a wrapper around g.findfile, which in turn is a wrapper around G_find_file2(), which is what C code would use for that purpose). -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Parser checking output maps but not input maps
Vaclav Petras wrote: is there some reason for GRASS parser not checking if the input map exists? It checks if the output map exists and if it it does it ends execution with an error. If an input map (or file) doesn't exist, you'll get an error when the module tries to open it. If I understand correctly, this applies only for C modules not Python or Bash modules, right? For scripts which use g.parser for argument parsing, if G_parser() fails g.parser itself fails, which in turn causes the script to fail. Python scripts should normally terminate on an exception if a spawned command fails, whereas shell scripts normally ignore the status of any spanwed commands. In both cases the error is not associated with the given option while in case of overwrite it is (which is nice). Correct. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] what is the meaning of: Error reading raster data for row 239 of MASK
Moritz Lennert wrote: The error comes from here: raster/get_row.c:142: G_fatal_error(_(Error reading raster data for row %d of %s), which is part of the 'read_data_compressed' function. read(fcb-data_fd, cmp, readamount) returns 0, while readamount is 18. I'm not familiar enough with the raster lib to understand what this implies. The odd thing is that a MASK raster is read using the same functions which are used for reading any other raster. Is the MASK raster itself fine? If you rename it to something other than MASK, can it be read by r.* commands without error? -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Parser checking output maps but not input maps
Vaclav Petras wrote: is there some reason for GRASS parser not checking if the input map exists? It checks if the output map exists and if it it does it ends execution with an error. If an input map (or file) doesn't exist, you'll get an error when the module tries to open it. If an output map exists, it would just get silently overwritten if it wasn't for the checks performed by G_parser(). -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Question on C API and GRASS raster/region
Pietro wrote: As you can see the region struct has now 43 rows and 47 columns, but still the Rast_window_rows and the Rast_window_cols return still the 'old' one. The Rast_window_[rows|cols] functions are described in the documentation () as Number of [rows|cols] in active window. with active here we mean current, aren't we? http://grass.osgeo.org/programming7/raster_2window_8c.html#ac2ce884f9f92cf60e28850c705ba8175 Therefore why they are not returning 43 and 47? You need to use Rast_set_window() (or Rast_set_input_window() and/or Rast_set_ouput_window()) to change the window(s) used by the raster library. libgis functions (e.g. G_set_window()) can't do this because libgis is at a lower level than libraster. Creating a new struct and asking for the current region provide me the expected result. Bear in mind that this won't affect the raster I/O functions. The raster windows are initialised from the libgis window at startup, but won't change thereafter unless changed via the libraster functions. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Debugging QGIS with GRASS provider on Windows
Helmut Kudrnovsky wrote: Finally I found it, it is similar story like with FILE. The provider (MSVC) calls Vect__open_old with struct Map_info variable allocated in the provider where sizeof(struct Map_info) = 1408. Vect__open_old (MinGW) calls G_zero on that variable, where sizeof(struct Map_info) = 1520. It means that all structures used in GRASS libs must be also allocated in GRASS. New functions like Vect_alloc_map have to be added to GRASS and until it gets to GRASS and to OSGeo4w, the the structures must be be allocated in the provider with enough space. Strange thing that it came up only now with GRASS 7 and threads. You need to get both compilers to use the same ABI. The alternative is to treat all structures as opaque, which isn't going to happen. It isn't sufficient to have GRASS allocate and free everything, you also can't access any fields directly, you'd need setters and getters for everything. Given the numbers, my first guess is that MinGW is using 8 bytes for off_t versus 4 bytes for MSVC. Yep; the bottom of config.h.in has: #if defined(__MINGW32__) /* add/remove as needed */ /* redefine off_t */ #include sys/types.h #define off_t off64_t And that's unaffected by --disable-largefile. This specific issue may just need changing __MINGW32__ to WIN32 (or _WIN32 or _WINNT, all of which are defined by MinGW) so that MSVC gets the same treatment. Failing that, some way to disable LFS (e.g. making MinGW builds honour --disable-largefile) is needed. The FILE issues are presumably due to mismatches between different versions of the MSVCRT library. malloc() and free() have the same issue: heap data must be freed by the same version of MSVCRT which allocated it (NVIZ originally had issues with passing Tcl_Alloc()d pointers directly to free() instead of Tcl_Free(), which fails if Tcl uses a different version of MSVCRT to GRASS). I don't know whether it's possible to change the version of MSVCRT which MinGW uses. If not, then we'll need to look at why files opened by lib/gis functions are being read/written by user code. But it would be preferable if FILE* could be passed around. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] how to work around Arument list too long error in a GRASS python script ?
Moritz Lennert wrote:
In a python script I have the following call:
grass.run_command('r.series',
input = rate_maps,
output = sum_rates,
method = 'sum',
overwrite = True,
quiet=True)
rate_maps is a list which in one instance contains 8559 map names,
leading to an OSError: [Errno 7] Argument list too long.
I know that in the shell I could use xargs to work around such a
problem. But how to do this in python ?
What it the OS limit for it?
I suppose this is ARG_MAX ?
getconf ARG_MAX
2097152
A text file with all file names only uses 144551 bytes.
Or is there another limit I should look at ?
Apparently there's a limit on the maximum length of a single argument,
which appears to be fixed at 128 KiB:
subprocess.call([echo, x*131071],stdout=devnull,stderr=devnull)
0
subprocess.call([echo, x*131072],stdout=devnull,stderr=devnull)
Traceback (most recent call last):
[snipped]
OSError: [Errno 7] Argument list too long
This specific issue could be worked around by having
grass.make_command() use multiple arguments when the value is a list,
e.g. r.series input=map1 input=map2
But such cases are likely to be problematic on other platforms where
ARG_MAX may be much lower. POSIX only requires that it is at least
4096, and it wasn't so long ago that such a value was commonplace. And
that value includes the environment as well as the command line.
Modules which commonly use many input maps should have the option to
read map names from a file. This mostly affects r.series, which
already does so.
If this issue was more common, we could consider extending G_parser()
to allow transparently reading any option value from a file, using
e.g. input=@filename. But that may just cause similar issues with
--interface-description, G_recreate_command(), etc.
--
Glynn Clements gl...@gclements.plus.com
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r65348: Mapset's tmp dir placement and broken element search
Martin Landa wrote: There is no justification for the current behaviour. The temporary cell/fcell and null files should always be created in the mapset's temporary directory. I am not sure why it must be so on (if files are not on the same filesystem, they can be copied), A copy is very different from rename(). It's not just about performance for performance' sake, but for minimising the interval when maps are corrupt (i.e. files exist in an inconsistent state). anyway I changed it in r65436. GRASS_TMPDIR_MAPSET is accepted only by vector library. Martin Martin Landa wrote: the question is whether to rename this variable to GRASS_VECTOR_TMPDIR_MAPSET. Any opinion? Martin Allowing normal tempfiles to be placed elsewhere is reasonable enough. But G_tempfile() originally existed specifically for the tempfile-and-rename idiom, and some code requires that behaviour (there are many more cases which ought to use that but don't). Given that misuse of G_tempfile() (i.e. using it like tmpnam()) is now more common than its intended use, it wouldn't be unreasonable to make G_tempfile() use the new behaviour and use a different name for the previous behaviour. The only requirement is that some function continues to provide the previous behaviour -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r65348: Mapset's tmp dir placement and broken element search
Vaclav Petras wrote: I am not sure why it must be so on (if files are not on the same filesystem, they can be copied), anyway I changed it in r65436. GRASS_TMPDIR_MAPSET is accepted only by vector library. Martin I don't understand why it should be OK for vectors when it is not OK for rasters. For rasters, it's a regression. The cell/fcell and null files used the tempfile-and-rename idiom from at least GRASS 4.3 (the oldest version I have) until the last week. There are plenty of other files which could ideally use the same mechanism, but as they're much smaller (and faster to write generally, as they tend to be simply dumps of existing data), it's less of an issue. Anyway, I think that non-atomicity during copy (instead of move) when requested (by GRASS_TMPDIR_MAPSET) is not much worse than anything else what one must deal with when using GRASS on some advanced configuration. For example, if you are writing to several vector maps with attributes in parallel, it will take forever (with SQLite). However, if you use PostgreSQL (or probably also DBF) as attribute backend, you get great performance. Does this mean that you cannot write vectors with attributes in parallel and perhaps also rasters to be on the safe side? No, it just means that you need to configure your environment according to what you are doing. Database updates should be handled using transactions where available. I don't know whether this is already the case; I'm not that familiar with the vector or DBMI libraries. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r65348: Mapset's tmp dir placement and broken element search
Vaclav Petras wrote: First, it allows unorthodox behavior of GRASS but this behavior is (or at least should be) applied only when GRASS_TMPDIR_MAPSET is set to some specific value. I don't see much problem with it as long as the documentation properly discuss all the potential issues and the default is the standard and safe way. This was discussed in [GRASS-dev] r65348 (GRASS_TMPDIR_MAPSET). There is no justification for the current behaviour. The temporary cell/fcell and null files should always be created in the mapset's temporary directory. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r65348 (GRASS_TMPDIR_MAPSET)
Martin Landa wrote: 2015-06-04 16:08 GMT+02:00 Glynn Clements gl...@gclements.plus.com: The the temporary files used by lib/raster need to be on the same filesystem as the mapset regardless of any environment settings. please could you write us why? So that they can be rename()d into place. Copy-and-delete isn't remotely the same thing. Aside from the gratuitous performance hit, it creates a window where the cell/fcell file is invalid. [Technically there's already a window because we remove() the file first because Windows doesn't support atomic rename(), but that's minuscule compared to the time taken to copy a file.] On that note, G_rename() should be called e.g. G_rename_or_copy() or G_move_file() or something, to avoid misleading anyone. A rename has specific semantics which G_rename() doesn't provide (e.g. preservation of hard links, permissions, ownership, extended attributes, inode number, holes, open descriptors, ... everything except the path, in fact). I didn't find any problem (bearing in mind changes in G_rename_file() which tries to rename file and if fails copy remove). Anyway, I wonder how to treat raster lib and vector lib differently. Martin I don't know what the vector library's requirements are, but any temporary file which is intended to be moved to a specific location should be created such that it can be rename()d to that location (i.e. on the same filesytem). The new behaviour is fine for code which wants a temporary file and doesn't care where it goes (e.g. because it will never be moved). But there's no justification for the previous behaviour being removed. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r65348 (GRASS_TMPDIR_MAPSET)
Martin Landa wrote: Yes. I'd prefer that lib/raster continues to create the file on the same filesystem as the mapset and atomically rename()s it into place. IOW, the new tempfile behaviour needs to be availble in addition to the long-standing behaviour, not a replacement for it. this is controlled by GRASS_TMPDIR_MAPSET variable, if not set the default is the long-standing behaviour. The the temporary files used by lib/raster need to be on the same filesystem as the mapset regardless of any environment settings. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r65348 (GRASS_TMPDIR_MAPSET)
Martin Landa wrote: I have updated G_rename_file() to deal with it [3], do you prefer another solution? Yes. I'd prefer that lib/raster continues to create the file on the same filesystem as the mapset and atomically rename()s it into place. IOW, the new tempfile behaviour needs to be availble in addition to the long-standing behaviour, not a replacement for it. Whether G_tempfile() itself has the old behaviour or the new behaviour isn't important. Given that most users of G_tempfile() don't care where the tempfile goes, making G_tempfile() have the new behaviour and using a different name for what lib/raster needs is probably less work. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] r.local.relief G_realloc error
Markus Neteler wrote: does anyone have an idea? https://gis.stackexchange.com/questions/148212/grass-make-local-relief-error r.contours cannot handle this amount of cells? r.contour creates a DCELL array for the current region then reads the raster data into it. 12500 rows by 1 cols by 8 bytes per cell is 1 GB of memory, on top of anything else the module uses. It's not inconceivable that the system's configured resource limits don't allow a single process to use that much. In such a situation, the first thing to do is to check the current soft limits with ulimit -a. If they are insufficient, check the hard limits with ulimit -aH. If the soft limits can be increased, do so; otherwise, ask the system administrator to increase the hard limits (these are normally set by the pam_limits module according to /etc/security/limits.conf and/or /etc/security/limits.d/*). Soft limits are usually set below the hard limits to prevent a misbehaving process from rendering the system unusable. On a multi-user system, hard limits may be set to prevent a single user from monopolising resource; there isn't much reason to set hard limits on a single-user system. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] compilation error - macro _n()
Martin Landa wrote: I'll let release manager to decide if it is safe to do so and just backport if it is. @Glynn: please do you have any opinion? If the macro and everything in the GRASS source tree have been changed, the only issue would be if any out-of-tree code (add-ons or private code) were using it. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [GRASS-SVN] r65264 - grass/trunk/lib/cairodriver
Vaclav Petras wrote: Unfortunately knowing that doesn't change for me that d.mon cairo output=vectormap.svg width=800 height=800 d.rast elevation looks nicer then export GRASS_RENDER_IMMEDIATE=cairo export GRASS_RENDER_FILE=vectormap.svg export GRASS_WIDTH=800 export GRASS_HEIGHT=800 d.rast elevation even for scripting. Whether it looks nicer isn't the issue. The d.mon example will overwrite the user's display settings. Write a d.mon-based example that preserves the user's display settings, then see which one looks nicer. Bonus marks if it correctly handles the case where the script aborts if any command fails. There were two main reasons why GRASS 7 abandoned the monitor concept, and one of them is that its statefulness made it next-to-impossible to write *robust* scripts involving display commands. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] clean_temp.c versus deleting mapset's .tmp
Vaclav Petras wrote: when I was working on grass.py I saw there usage of clean_temp function which calls clean_temp program and deletion of the whole .tmp directory and I can't make sense out of it. It seems to me that clean_temp (lib/init/clean_temp.c) executable called in grass.py is not necessary because the whole .tmp directory is anyway deleted at the beginning and at the end of the session. clean_temp is doing some special cleaning according to pids but whole directory is deleted anyway at the end. Can the whole lib/init/clean_temp.c can be safely removed? Or maybe it has some meaning when executed before the session starts because at that point .tmp is not deleted. Can someone please clarify this for me here or in the source code? Also, the clean_temp function is called manually while other cleanups are called using at exit mechanism. Supposing we need the clean up at all, I'm not sure if there is any reason to keep it explicit. Perhaps this was just inconsistency, not an intention. clean_temp is more restrictive in what it will remove (only files owned by the current user, only files older than 4 days). The original rationale for this behaviour is probably forgotten, and it's unclear if it's still valid (probably not, if the startup script is removing the entire directory at the end of the session and no-one has complained). -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] [GRASS-SVN] r65264 - grass/trunk/lib/cairodriver
Vaclav Petras wrote: I think this is exactly what I mean. Any reason for using the variables instead of d.mon? d.mon isn't something you'd want to use from scripts or other programs. Ultimately, d.mon is just causing those environment variables to be set whenever you run a display command, so documenting the variables themselves is more general than documenting a specific mechanism used to set them. And if you don't need the convenience of the d.mon interface, setting the variables directly removes some complexity from the process. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] compilation error - macro _n()
Maris Nartiss wrote: As it is used by GRASS internally only, a simple sed (+ documentation change) should be enough. The only question remains - change to what? n_()? -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] compilation error - macro _n()
Martin Landa wrote:
recently I got problem compiling GRASS module against CGAL library.
The source of problem is duplicated definition of _n() macro.
In file included from /usr/include/boost/random.hpp:37:0,
from /usr/local/include/CGAL/spatial_sort.h:31,
from /usr/local/include/CGAL/Triangulation_3.h:47,
from local_proto.h:4,
from main.cpp:25:
/usr/include/boost/random/discard_block.hpp:60:42: error: macro _n
requires 3 arguments, but only 1 given
discard_block_engine() : _rng(), _n(0) { }
GRASS defines _n() macro as [1], Boost is using _n() macro in different
meaning.
Any idea who to fix this problem correctly?
Boost isn't using any _n() macro; that's the C++ syntax for
initialising a member variable named _n upon construction.
At least, it isn't trying to use a macro. The fact that there happens
to be a macro called _n() in existence at that point is beyond its
control.
It's GRASS' fault for using a reserved name (_n() was added in r59156,
r59189, now 137 occurrences in 52 files), and particularly for using
it as a macro.
Including the CGAL headers before the GRASS headers would avoid this
specific issue.
Note that it's fairly common for C++ code to use such names within
class scope because C++ *has* class scope and reserved names aren't
reserved there. Names created in class scope don't affect other scopes
(and aren't affected by other scopes), and using a reserved name would
normally help to avoid this exact issue.
Macros aren't scoped, so the only way to avoid having a macro
definition breaking your code is to choose a name that's unlikely to
be used as a macro. Ordinarily, a name that's reserved to the
implementation in any scope (i.e. prohibited from being used as a
macro) would be a safe bet for that.
--
Glynn Clements gl...@gclements.plus.com
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] gis.py:801: DeprecationWarning: classic int division
Nikos Alexandris wrote:
I just ran
python -Qwarn -tt -3
on a custom python script and the first line returned is
gis.py:801: DeprecationWarning: classic int division
Is this something that needs to be fixed?
That specific warning relates to code which is generated by ctypesgen
from system headers. Specifically, the definition of __sigset_t in
sigset.h:
# define _SIGSET_NWORDS (1024 / (8 * sizeof (unsigned long int)))
typedef struct
{
unsigned long int __val[_SIGSET_NWORDS];
} __sigset_t;
is translated to:
# /usr/include/bits/sigset.h: 30
class struct_anon_11(Structure):
pass
struct_anon_11.__slots__ = [
'__val',
]
struct_anon_11._fields_ = [
('__val', c_ulong * (1024 / (8 * sizeof(c_ulong,
]
__sigset_t = struct_anon_11 # /usr/include/bits/sigset.h: 30
The reason this is ending up in gis.py is for the definition of
jmp_buf which is used by G_fatal_longjmp().
As that function is (probably) unusable from Python, we could probably
just guard the declaration with #ifndef CTYPESGEN.
But that may not be the only such issue.
For compatibility with Python 3, we need to add
from __future__ import division
and use // for integer (truncating) division. But IIRC, that
requires Python 2.7. If GRASS still works with 2.6, it's not worth
abandonding support for it over this issue, given that there are
almost certainly far more significant issues with Python 3
compatibility (such as its insistence on coercing everything to
Unicode).
--
Glynn Clements gl...@gclements.plus.com
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Limit in g.message? was: Limit in number of variables for r.mapcalc?
Markus Neteler wrote: But more significantly, G_message() etc use an internal buffer of 2000 bytes. A message longer than that probably won't work and may well cause the g.message to crash. How about (ab)using GPATH_MAX (include/gis.h) which is 4096 chars long and using it in general/g.message/main.c as well to avoid a buffer overflow? Having g.message limit the length of a message would work for g.message. For G_message() etc, using vsnprintf() would work, but that's C99. G_vasprintf() should work, but our private implementation (used if asprintf() doesn't exist) also requires C99. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Limit in g.message? was: Limit in number of variables for r.mapcalc?
Nikos Alexandris wrote: So the question is, is there a limit in g.message? Number of characters? g.message takes the message via a command-line argument, so it's limited to any system-imposed limit on the length of an individual argument or the combined length of the arguments. But more significantly, G_message() etc use an internal buffer of 2000 bytes. A message longer than that probably won't work and may well cause the g.message to crash. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
