Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Glynn Clements wrote:
Markus Metz wrote:
Considering this, rather use e.g. raster_min - 1 as default nodata value
for GDAL floating point datatypes?
If fabs(raster_min) is large, raster_min - 1 won't be exactly
representable, and may be rounded to raster_min. You would need to
subtract a value which depends upon the exponent.
Also, the input data may contain -DBL_MAX. For FCELL, if the map is
large enough, it's possible that *all* finite values occur in the
data.
I was (honestly! I was listening to you:-)) thinking about something like
if (raster_min type_min) {
unsigned int i = 0;
while ((float)(raster_min - ++i) == raster_min);
nodata = raster_min - i;
}
else if (raster_max type_max) {
unsigned int i = 0;
while ((float)(raster_max + ++i) == raster_max);
nodata = raster_max + i;
}
else
nodata = 0; /* or whatever fallback */
for GDT_Float32 but did not want to go into so much detail.
The question is: is GDAL's no-data value allowed to be any FP value,
or must it be finite? I'm guessing that the GDAL documentation doesn't
say (similarly, I don't think it's documented whether GRASS
FCELL/DCELL values are allowed to be infinite).
Actually, this may depend upon the format. If a format supports FP,
the format's specification may dictate whether or not NaN and/or
inifinities are considered valid.
Hmm. I thought the point of using GDAL is that r.out.gdal does *not*
have to bother about the format's specifications. IMHO, if there is
reason to assume that NaN and/or infinities are not valid for any of the
formats supporting FP, r.out.gdal should not use NaN and/or infinities.
OTOH, NaN and infinities are described in the IEEE FP standard. I would
avoid format-specific code in r.out.gdal, considering the large number
of supported formats and the number of creation options for each format
(for each format sometimes differing between datatypes, e.g. GeoTIFF).
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Glynn Clements wrote: Markus Metz wrote: - all floating point: IEEE's NaN Problem with NaN? According to IEEE 754, x == y is always FALSE if either x or y or both are NaN. Assuming (nodata = GDALGetRasterNoDataValue()) == NaN, Sorry, should have been nodata = GDALGetRasterNoDataValue() and isnan(nodata) then going through all cells if (cell == nodata) will always be FALSE, nodata cells may not be detected? Depends on each implementation of nodata detection? Can be solved for GRASS, but how is this done in all other GIS applications? if (x == nodata || isnan(nodata) isnan(x)) isnan() is C99 and POSIX, or it can be defined as e.g.: #define ISNAN(x) ((x) != (x)) As previously in ticket #73, I created a MASK and then exported elevation from nc_spm_08 as FLOAT32, once with nodata=0 and once with default nodata value NaN for GDT_Float32. GRASS 7 r.in.gdal: all ok GRASS 7 r.external: all ok Other GIS applications: QGIS 1.0.0: all ok gvSIG 1.1.2: doesn't support the nodata concept, and NaN is displayed like 0.0 SAGA 2.0.3: display ok for NaN, but cell query gives 0.0 instead of NaN or nodata, and NoData info says 0 ESRI ArcMap 9.2: display ok for nodata=0, not ok when using NaN, NaN displayed like value 0.0, and cell query gives NaN instead of nodata Considering this, rather use e.g. raster_min - 1 as default nodata value for GDAL floating point datatypes? ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Markus Metz wrote: Considering this, rather use e.g. raster_min - 1 as default nodata value for GDAL floating point datatypes? If fabs(raster_min) is large, raster_min - 1 won't be exactly representable, and may be rounded to raster_min. You would need to subtract a value which depends upon the exponent. Also, the input data may contain -DBL_MAX. For FCELL, if the map is large enough, it's possible that *all* finite values occur in the data. The question is: is GDAL's no-data value allowed to be any FP value, or must it be finite? I'm guessing that the GDAL documentation doesn't say (similarly, I don't think it's documented whether GRASS FCELL/DCELL values are allowed to be infinite). Actually, this may depend upon the format. If a format supports FP, the format's specification may dictate whether or not NaN and/or inifinities are considered valid. -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Glynn Clements wrote: Markus Metz wrote: so the user will have to understand the conversion issues even if they never use an out-of-range value. Ultimately, its either usability or flexibility. Without the -f flag I would opt for usability, with the -f flag for flexibility. Having -f affect the interpretation of nodata= also harms usability. I know. There was discussion about that, and Dylan asked for the possibility to 2. include a 'force' option to make r.out.gdal do exactly what your command line instructions suggest-- possibly leading to corruption of NULL data Maybe this referred only to the problem of input data having the requested nodata value, and not to allow a mismatch between metadata nodata and the value replacing NULL cells? The GDAL API documentation says To clear the nodata value, just set it with an out of range value. This is currently possible with the -f flag, but r.out.gdal then does not give any information on what value is assigned to NULL cells, only what nodata value is written to the metadata. Right. There's only one nodata= option, and it's performing a dual function: what to store for null cells and what to set GDAL's no-data value to. If you want to allow a mismatch, you really need two options. I thought in this case (allow mismatch) it would be enough to write it as double to metadata without prior cast to the selected GDAL datatype. Yeah, but what are you going to write for nulls in the input data? Suppose the user specifies a CELL input map (which contains at least 0-255 and null), byte (GDT_UInt8) output, -f, and nodata=. So you use .0 as GDAL's no-data value, but how will you treat null values in the input data? In this case, the current code of all C versions of r.out.gdal writes (CELL) for NULLs. This is then cast by the gdal library to GDT_Byte (255). Maybe I misunderstood the requested behaviour for the -f flag, so let's not allow a nodata mismatch, convert the nodata value first to the selected GDAL datatype if necessary and then we get the same value for nodata metadata and raster data where NULLs were replaced. In this case, there should be a warning because (GDT_Byte) .0 = 255 is present in the input data and the user is asked to select a different nodata value. Export would proceed because of -f. Anyway, now I will stop defending a feature that I was never convinced of (allowing this nodata mismatch). Can we now put together a next version of r.out.gdal based on this discussion that always interprets any user-given nodata value (with warning where appropriate) to make sure metadata info and raster data are the same? ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
On 22/04/09 09:51, Markus Metz wrote: Anyway, now I will stop defending a feature that I was never convinced of (allowing this nodata mismatch). Can we now put together a next version of r.out.gdal based on this discussion that always interprets any user-given nodata value (with warning where appropriate) to make sure metadata info and raster data are the same? +1 Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Markus Metz wrote: - all floating point: IEEE's NaN Problem with NaN? According to IEEE 754, x == y is always FALSE if either x or y or both are NaN. Assuming (nodata = GDALGetRasterNoDataValue()) == NaN, then going through all cells if (cell == nodata) will always be FALSE, nodata cells may not be detected? Depends on each implementation of nodata detection? Can be solved for GRASS, but how is this done in all other GIS applications? if (x == nodata || isnan(nodata) isnan(x)) isnan() is C99 and POSIX, or it can be defined as e.g.: #define ISNAN(x) ((x) != (x)) -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Glynn Clements wrote: Markus Metz wrote: I think I slowly begin to understand. You suggest to use (GDT_Int32)0x8000 for both the GDAL metadata info and the GDAL raster data although GDAL metadata nodata is double? Yes; for the latter, the value would be: (double) (GDT_Int32) 0x8000 IOW, -2147483648.0. Note that this isn't the same as atof(0x8000), which would be positive. Obviously. Because of the request for a -f flag I was fixed on leaving r.out.gdal parameters as they are, also leaving default nodata as double, no prior cast to the GDAL datatype. My confusion, it's clear now what you want for the default nodata value. Further on, I understand you suggest to interpret the user given nodata option by writing to metadata (double) (GDAL datatype) nodata value. No more objections from my side, but I would print a warning if (nodata != (double) (GDAL data type) nodata), telling the user what the nodata value will be after GDAL converted it. I would suggest to keep writing (double) nodata value with the -f flag, no prior cast. In case of the -f flag and a mismatch (nodata != (double) (GDAL data type) nodata), a message/warning could be printed with both the nodata value written to metadata and the value assigned to NULL cells. I was assuming that not only CELL, but also FCELL and DCELL maps may be exported as GDT_UInt32. Then 2^31 can occur in the raster to be exported. Actually, I was assuming that any GRASS raster type may be exported as any GDAL data type. Right. But exporting FCELL or DCELL as GDT_UInt32 is inherently lossy. Just to make it clear: I don't insist on exporting to GDT_UInt32, but this is supported by r.out.gdal for all GRASS raster types, so this possibility must be considered IMHO, unless GDT_UInt32 becomes disabled. For DCELL, every possible GDT_UInt32 value may occur in the source data, so there isn't any ideal default no-data value. That's why I first want to check the range of the raster map(s) to be exported. What if the user asks for a nodata value to be out of range, i.e. really wants it to be out of range, leading to a mismatch between the nodata value in the metadata and the value assigned to NULL cells in the GDAL export? I understood that this was requested to be possible. Hmm. To allow that, you can't cast to the source or destination type, Exactly. That's why I was thinking about e.g. (double) 2147483648 to be written to metadata, and not (double) (GDT_Int32) 2147483648. BTW, I can still not see why a user would want that, but it can be allowed. so the user will have to understand the conversion issues even if they never use an out-of-range value. Ultimately, its either usability or flexibility. Without the -f flag I would opt for usability, with the -f flag for flexibility. If r.out.gdal should support out of range nodata values, and the nodata value is read with atof(), 2^31 should stay 2^31and not be cast to -2^31. The GUI description of the nodata option says that it is of type float (actually it's a double), i.e. suggesting that any value that is representable as floating point is ok. That's a limitation of the parser. If the input is CELL, then the nodata value which the input data uses isn't any kind of float. It might be more correct to have separate int/float nodata options. OTOH, GDAL's no-data value is always a double, even for integer data. I would rather stick to GDAL's method of handling nodata options and try to mimic that in r.out.gdal. I don't have any preference, although I suspect that users will regularly get bitten by the conversion issues. As above, by default r.out.gdal could write (double) (GDAL datatype) nodata value to metadata, with the -f flag no cast to GDAL datatype? The GDAL API documentation says To clear the nodata value, just set it with an out of range value. This is currently possible with the -f flag, but r.out.gdal then does not give any information on what value is assigned to NULL cells, only what nodata value is written to the metadata. Right. There's only one nodata= option, and it's performing a dual function: what to store for null cells and what to set GDAL's no-data value to. If you want to allow a mismatch, you really need two options. I thought in this case (allow mismatch) it would be enough to write it as double to metadata without prior cast to the selected GDAL datatype. I don't see the point in treating range limits differently to precision limits. Either the output type is a superset of the input type (lossless export) or it isn't (lossy export). Easy to change, with feedback to the user as warning in case of lossy export. When exporting to FP types, the default should probably be NaN (if supported), regardless of whether the source is integer or FP. Exporting CELL to GDT_Float32 should print a warning if the range of the input data exceeds +/- 2^24, as it cannot be
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Markus Metz wrote: so the user will have to understand the conversion issues even if they never use an out-of-range value. Ultimately, its either usability or flexibility. Without the -f flag I would opt for usability, with the -f flag for flexibility. Having -f affect the interpretation of nodata= also harms usability. The GDAL API documentation says To clear the nodata value, just set it with an out of range value. This is currently possible with the -f flag, but r.out.gdal then does not give any information on what value is assigned to NULL cells, only what nodata value is written to the metadata. Right. There's only one nodata= option, and it's performing a dual function: what to store for null cells and what to set GDAL's no-data value to. If you want to allow a mismatch, you really need two options. I thought in this case (allow mismatch) it would be enough to write it as double to metadata without prior cast to the selected GDAL datatype. Yeah, but what are you going to write for nulls in the input data? Suppose the user specifies a CELL input map (which contains at least 0-255 and null), byte (GDT_UInt8) output, -f, and nodata=. So you use .0 as GDAL's no-data value, but how will you treat null values in the input data? -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Glynn Clements wrote: Markus Metz wrote: OK, but I have tested r.out.gdal with type = GDT_Int32 and nodata = 0x8000. Same result: NULL cells become -2147483648 but the nodata value in the metadata says 2147483648 (gdalinfo on the export output), Yes. I know. Running r.out.gdal ... nodata=0x8000 is misinterpreting what I was saying, as r.out.gdal parses nodata= as a floating-point value. I'm talking about using (GDT_Int32)0x8000 (i.e. -2^31) I think I slowly begin to understand. You suggest to use (GDT_Int32)0x8000 for both the GDAL metadata info and the GDAL raster data although GDAL metadata nodata is double? When using a default nodata value, what I want to get in the exported map is identical values for the metadata nodata value and the value assigned to NULL cells. GDAL metadata nodata value is always double (GDALSetRasterNoDataValue()), the value assigned to NULL cells is of the selected GDAL datatype. So 0x8000 for GDT_Int32 will be 2^31 in metadata and -2^31 in the data if passed as such to the corresponding GDAL functions, (double ) 0x8000 for metadata info and (GDT_Int32) 0x8000 for raster data. or (GDT_UInt32)0x8000 (i.e. +2^31) as appropriate for the destination type. I was assuming that not only CELL, but also FCELL and DCELL maps may be exported as GDT_UInt32. Then 2^31 can occur in the raster to be exported. Actually, I was assuming that any GRASS raster type may be exported as any GDAL data type. For r.out.gdal, there was discussion about not to override user options and instead issue an error or a warning (going for error now) that the nodata value is out of range. This isn't about overriding user options, but interpreting them correctly. Close to the module knows better which is not desirable according to Dylan and Moritz. No, the opposite. Writing code which does what the user asks for rather than doing something else which is easier to code. What if the user asks for a nodata value to be out of range, i.e. really wants it to be out of range, leading to a mismatch between the nodata value in the metadata and the value assigned to NULL cells in the GDAL export? I understood that this was requested to be possible. If r.out.gdal should support out of range nodata values, and the nodata value is read with atof(), 2^31 should stay 2^31and not be cast to -2^31. The GUI description of the nodata option says that it is of type float (actually it's a double), i.e. suggesting that any value that is representable as floating point is ok. That's a limitation of the parser. If the input is CELL, then the nodata value which the input data uses isn't any kind of float. It might be more correct to have separate int/float nodata options. OTOH, GDAL's no-data value is always a double, even for integer data. I would rather stick to GDAL's method of handling nodata options and try to mimic that in r.out.gdal. OTOH, reading as FP allows you to specify +2^31 as the nodata value when exporting CELL as UInt32. Maybe it would be better to read as double then cast to the destination type. Hmm, then it would be no longer possible to have an out-of-range nodata value in the metadata. I would support that solution, but it was explicitly requested to obey all user options when the -f flag is used, also writing the nodata value as it is to the metadata. It is then up to the user to figure out what happened to the NULL cells, assuming that the user knows what happens when using the -f flag. I can see some point to that, but may be confusing for the user, who may assume that they're talking about the data's no-data value when they're actually specifying GDAL's no-data value. That's why there are warnings if there will be a mismatch between the GDAL metadata nodata value and the GDAL raster nodata value (out of range test). And having this mismatch is only possible with the -f flag, when forcing r.out.gdal to use exactly the given parameters and options. The GDAL API documentation says To clear the nodata value, just set it with an out of range value. This is currently possible with the -f flag, but r.out.gdal then does not give any information on what value is assigned to NULL cells, only what nodata value is written to the metadata. Even so, 0x8000 is still the ideal value for exporting CELL data to either UInt32 or Int32. You just have to pick the correct interpretation of it (unsigned int or signed int respectively). The current as well as previous policy of r.out.gdal is to not to interpret the nodata value. It is written as is (currently double) to metadata, same like e.g. gdal_translate -a_nodata behaves. It doesn't have any choice about interpreting it. nodataopt-answer is a char*, while GDALSetRasterNoDataValue() expects a double. There are several ways to convert a char* to a double; atof() may be the simplest to implement, but is it the most correct? No idea. If you can't think if a more correct way,
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Markus Metz wrote: OK, but I have tested r.out.gdal with type = GDT_Int32 and nodata = 0x8000. Same result: NULL cells become -2147483648 but the nodata value in the metadata says 2147483648 (gdalinfo on the export output), Yes. I know. Running r.out.gdal ... nodata=0x8000 is misinterpreting what I was saying, as r.out.gdal parses nodata= as a floating-point value. I'm talking about using (GDT_Int32)0x8000 (i.e. -2^31) I think I slowly begin to understand. You suggest to use (GDT_Int32)0x8000 for both the GDAL metadata info and the GDAL raster data although GDAL metadata nodata is double? Yes; for the latter, the value would be: (double) (GDT_Int32) 0x8000 IOW, -2147483648.0. Note that this isn't the same as atof(0x8000), which would be positive. When using a default nodata value, what I want to get in the exported map is identical values for the metadata nodata value and the value assigned to NULL cells. GDAL metadata nodata value is always double (GDALSetRasterNoDataValue()), the value assigned to NULL cells is of the selected GDAL datatype. So 0x8000 for GDT_Int32 will be 2^31 in metadata and -2^31 in the data if passed as such to the corresponding GDAL functions, (double ) 0x8000 for metadata info and (GDT_Int32) 0x8000 for raster data. That depends upon how you interpret 0x8000. (double) (GDT_Int32) 0x8000 is equal to -2147483648.0, while atof(0x8000) is equal to +2147483648.0. GDT_Int32 itself cannot represent +2^31. When writing CELL data as GDT_Int32, the default null value should be (GDT_Int32) 0x8000, as that's the only value which cannot occur in the source data. The GDAL no-data value should be that value cast to a double, i.e.: (double) (GDT_Int32) 0x8000 or: atof(-0x8000) or: -2147483648.0 or any other permutation which produces the same double value. or (GDT_UInt32)0x8000 (i.e. +2^31) as appropriate for the destination type. I was assuming that not only CELL, but also FCELL and DCELL maps may be exported as GDT_UInt32. Then 2^31 can occur in the raster to be exported. Actually, I was assuming that any GRASS raster type may be exported as any GDAL data type. Right. But exporting FCELL or DCELL as GDT_UInt32 is inherently lossy. For DCELL, every possible GDT_UInt32 value may occur in the source data, so there isn't any ideal default no-data value. For r.out.gdal, there was discussion about not to override user options and instead issue an error or a warning (going for error now) that the nodata value is out of range. This isn't about overriding user options, but interpreting them correctly. Close to the module knows better which is not desirable according to Dylan and Moritz. No, the opposite. Writing code which does what the user asks for rather than doing something else which is easier to code. What if the user asks for a nodata value to be out of range, i.e. really wants it to be out of range, leading to a mismatch between the nodata value in the metadata and the value assigned to NULL cells in the GDAL export? I understood that this was requested to be possible. Hmm. To allow that, you can't cast to the source or destination type, so the user will have to understand the conversion issues even if they never use an out-of-range value. Ultimately, its either usability or flexibility. If r.out.gdal should support out of range nodata values, and the nodata value is read with atof(), 2^31 should stay 2^31and not be cast to -2^31. The GUI description of the nodata option says that it is of type float (actually it's a double), i.e. suggesting that any value that is representable as floating point is ok. That's a limitation of the parser. If the input is CELL, then the nodata value which the input data uses isn't any kind of float. It might be more correct to have separate int/float nodata options. OTOH, GDAL's no-data value is always a double, even for integer data. I would rather stick to GDAL's method of handling nodata options and try to mimic that in r.out.gdal. I don't have any preference, although I suspect that users will regularly get bitten by the conversion issues. OTOH, reading as FP allows you to specify +2^31 as the nodata value when exporting CELL as UInt32. Maybe it would be better to read as double then cast to the destination type. Hmm, then it would be no longer possible to have an out-of-range nodata value in the metadata. I would support that solution, but it was explicitly requested to obey all user options when the -f flag is used, also writing the nodata value as it is to the metadata. It is then up to the user to figure out what happened to the NULL cells, assuming that the user knows what happens when using the -f flag. I can see some point to that, but may be confusing for the user,
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Glynn Clements wrote: Markus Metz wrote: I guess a minimum requirement would be that something exported with r.out.gdal and then imported again with r.in.gdal should be identical (taking into consideration the region settings during export) and not have any data loss, be it due to clipping data or due to nodata problems. That's only possible if the exported data is in a format which can hold the full range of the data. A UInt32 TIFF can't hold the full range of CELL data (it can't hold any negative values). A UInt32 can hold the full range of a CELL raster map if there are no negative values. What you mean is the potential range, what I mean is the data range actually present in the raster. Which still means that UInt32 doesn't really make sense as GDAL export type for GRASS rasters. That would also mean that r.out.gdal with type = GDT_Int32 and nodata = 2147483648, NULL cells become -2147483648 but the nodata value in the metadata stays 2147483648 (gdalinfo on the export output), which in turn means that other software, also QGIS, does not see any nodata cells in the export output -information loss about NULL cells. That's why I wouldn't use 0x8000 as default nodata value for GDT_Int32 in r.out.gdal even if r.in.gdal has no problems with it. You're misinterpretating my use of 0x8000 here. I'm talking about the C interpretation, which is the (signed) integer value -2147483648. OK, but I have tested r.out.gdal with type = GDT_Int32 and nodata = 0x8000. Same result: NULL cells become -2147483648 but the nodata value in the metadata says 2147483648 (gdalinfo on the export output), and other software including QGIS doesn't see any nodata cells. I haven't changed the part of the code of r.out.gdal that writes the nodata value to metadata, also not the code that writes the actual raster band. For r.out.gdal, there was discussion about not to override user options and instead issue an error or a warning (going for error now) that the nodata value is out of range. This isn't about overriding user options, but interpreting them correctly. Close to the module knows better which is not desirable according to Dylan and Moritz. It's debatable whether just using atof(nodataopt-answer) directly is actually a correct interpretation when the input map is CELL, or whether the code should use e.g. atoi(nodataopt-answer) or (CELL)atof(nodataopt-answer) for CELL inputs. Reading as int (or casting to it) will interpret 0x8000 as -2^31, while reading as FP will interpret it as 2^31. If r.out.gdal should support out of range nodata values, and the nodata value is read with atof(), 2^31 should stay 2^31and not be cast to -2^31. The GUI description of the nodata option says that it is of type float (actually it's a double), i.e. suggesting that any value that is representable as floating point is ok. OTOH, reading as FP allows you to specify +2^31 as the nodata value when exporting CELL as UInt32. Maybe it would be better to read as double then cast to the destination type. Hmm, then it would be no longer possible to have an out-of-range nodata value in the metadata. I would support that solution, but it was explicitly requested to obey all user options when the -f flag is used, also writing the nodata value as it is to the metadata. It is then up to the user to figure out what happened to the NULL cells, assuming that the user knows what happens when using the -f flag. Currently, all default nodata values are within the range of the selected GDAL data type. User-defined nodata values are tested whether they are within the range of the selected data type, if not, r.out.gdal in trunk and devbr6 exits with an error message. r.out.gdal also exits with an error message if the nodata value is present in the raster to be exported in order to avoid data loss. This is a rather strict and conservative approach because of the reported problems with nodata in r.out.gdal exports. Right. Even so, 0x8000 is still the ideal value for exporting CELL data to either UInt32 or Int32. You just have to pick the correct interpretation of it (unsigned int or signed int respectively). The current as well as previous policy of r.out.gdal is to not to interpret the nodata value. It is written as is (currently double) to metadata, same like e.g. gdal_translate -a_nodata behaves. Afraid of rewriting large parts of r.out.gdal: can't we just use -2147483648 as default nodata value for Int32 and 0 as default nodata value for UInt32? Getting tired of this UInt32 example, but again: (unsigned int) 0x8000 is 2^31, yes? Then this value would be right in the middle of the range of UInt32, and can cause data loss when exporting a FCELL raster as UInt32 . For the sake of the argument, let's assume this FCELL raster has only positive values, some of them 2^31, and any errors when casting from FCELL to UInt32 are acknowledged and
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Markus Metz wrote: That would also mean that r.out.gdal with type = GDT_Int32 and nodata = 2147483648, NULL cells become -2147483648 but the nodata value in the metadata stays 2147483648 (gdalinfo on the export output), which in turn means that other software, also QGIS, does not see any nodata cells in the export output -information loss about NULL cells. That's why I wouldn't use 0x8000 as default nodata value for GDT_Int32 in r.out.gdal even if r.in.gdal has no problems with it. You're misinterpretating my use of 0x8000 here. I'm talking about the C interpretation, which is the (signed) integer value -2147483648. OK, but I have tested r.out.gdal with type = GDT_Int32 and nodata = 0x8000. Same result: NULL cells become -2147483648 but the nodata value in the metadata says 2147483648 (gdalinfo on the export output), Yes. I know. Running r.out.gdal ... nodata=0x8000 is misinterpreting what I was saying, as r.out.gdal parses nodata= as a floating-point value. I'm talking about using (GDT_Int32)0x8000 (i.e. -2^31) or (GDT_UInt32)0x8000 (i.e. +2^31) as appropriate for the destination type. For r.out.gdal, there was discussion about not to override user options and instead issue an error or a warning (going for error now) that the nodata value is out of range. This isn't about overriding user options, but interpreting them correctly. Close to the module knows better which is not desirable according to Dylan and Moritz. No, the opposite. Writing code which does what the user asks for rather than doing something else which is easier to code. It's debatable whether just using atof(nodataopt-answer) directly is actually a correct interpretation when the input map is CELL, or whether the code should use e.g. atoi(nodataopt-answer) or (CELL)atof(nodataopt-answer) for CELL inputs. Reading as int (or casting to it) will interpret 0x8000 as -2^31, while reading as FP will interpret it as 2^31. If r.out.gdal should support out of range nodata values, and the nodata value is read with atof(), 2^31 should stay 2^31and not be cast to -2^31. The GUI description of the nodata option says that it is of type float (actually it's a double), i.e. suggesting that any value that is representable as floating point is ok. That's a limitation of the parser. If the input is CELL, then the nodata value which the input data uses isn't any kind of float. It might be more correct to have separate int/float nodata options. OTOH, GDAL's no-data value is always a double, even for integer data. OTOH, reading as FP allows you to specify +2^31 as the nodata value when exporting CELL as UInt32. Maybe it would be better to read as double then cast to the destination type. Hmm, then it would be no longer possible to have an out-of-range nodata value in the metadata. I would support that solution, but it was explicitly requested to obey all user options when the -f flag is used, also writing the nodata value as it is to the metadata. It is then up to the user to figure out what happened to the NULL cells, assuming that the user knows what happens when using the -f flag. I can see some point to that, but may be confusing for the user, who may assume that they're talking about the data's no-data value when they're actually specifying GDAL's no-data value. Even so, 0x8000 is still the ideal value for exporting CELL data to either UInt32 or Int32. You just have to pick the correct interpretation of it (unsigned int or signed int respectively). The current as well as previous policy of r.out.gdal is to not to interpret the nodata value. It is written as is (currently double) to metadata, same like e.g. gdal_translate -a_nodata behaves. It doesn't have any choice about interpreting it. nodataopt-answer is a char*, while GDALSetRasterNoDataValue() expects a double. There are several ways to convert a char* to a double; atof() may be the simplest to implement, but is it the most correct? Afraid of rewriting large parts of r.out.gdal: can't we just use -2147483648 as default nodata value for Int32 Yes. and 0 as default nodata value for UInt32? No; 0 is a valid data value. Use +2147483648, as that will never occur in the input data. Or even any value = 2147483648. Getting tired of this UInt32 example, but again: (unsigned int) 0x8000 is 2^31, yes? Yes. Then this value would be right in the middle of the range of UInt32, and can cause data loss when exporting a FCELL raster as UInt32 . In that case, use (double) 0xU, which will never exist in CELL or FCELL data. Maybe the default should depend upon the input type? -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Moritz Lennert wrote: In the current state is there a possibility of using gdal's acceptance of unvalid nodata ? And can we force a nodata value which exists in the map ? I would agree with Dylan that this kind of brute-force method should remain possible, be it through the suggested -f flag. I'm always wary of programs that assume they know better than you what you want ;-) Added -f flag to force export in trunk r36769 and devbr6 r36770. With the -f flag, r.out.gdal overrides the new safety checks, but still warns if thinks there may be problems. Scripts will work as before, but raster export will be aborted if any of the safety checks is not passed (supposed to be a bugfix, not an enhancement). I guess this ticket can only be closed after backporting to 6.4. Markus M ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Markus Metz wrote:
This is now a mix of r.in.gdal and r.out.gdal. The two modules
complement each other, and I guess a minimum requirement would be that
something exported with r.out.gdal and then imported again with
r.in.gdal should be identical (taking into consideration the region
settings during export) and not have any data loss, be it due to
clipping data or due to nodata problems.
That's only possible if the exported data is in a format which can
hold the full range of the data. A UInt32 TIFF can't hold the full
range of CELL data (it can't hold any negative values).
The only fool-proof nodata value for CELL is 0x8000; all other
values may be actual data values.
For grass rasters of type CELL, not for gdal export as Int32. gdalinfo
says NoData Value=2147483648, but cells that should be nodata are
-2147483648 and are thus not recognized as nodata.
It worked as of a few days ago, in the version I have checked out:
raster/r.in.gdal/main.c:780:
if (((GInt32 *) cell)[indx] == (GInt32) dfNoData) {
(GInt32)2147483648 and (GInt32)-2147483648 are equal.
I think the problem is that gdal allows the nodata value to be out of
range with regard to the raster type,
Yes.
e.g. in gdal, a nodata value of
- with GDT_Byte is legal and can be used to effectively disable
nodata information. No idea if this conforms to GeoTIFF specs (probably
yes), but I'm pretty sure that this is the current policy of gdal
(possible with e.g. gdal_translate).
That would also mean that r.out.gdal with type = GDT_Int32 and nodata =
2147483648, NULL cells become -2147483648 but the nodata value in the
metadata stays 2147483648 (gdalinfo on the export output), which in turn
means that other software, also QGIS, does not see any nodata cells in
the export output -information loss about NULL cells. That's why I
wouldn't use 0x8000 as default nodata value for GDT_Int32 in
r.out.gdal even if r.in.gdal has no problems with it.
You're misinterpretating my use of 0x8000 here. I'm talking
about the C interpretation, which is the (signed) integer value
-2147483648.
If it's been changed, use nodata=-0x8000.
That's currently done in r.out.gdal in trunk and devbr6.
Or cast the nodata value
to the source data type then back to double.
For r.out.gdal, there was discussion about not to override user options
and instead issue an error or a warning (going for error now) that the
nodata value is out of range.
This isn't about overriding user options, but interpreting them
correctly.
It's debatable whether just using atof(nodataopt-answer) directly is
actually a correct interpretation when the input map is CELL, or
whether the code should use e.g. atoi(nodataopt-answer) or
(CELL)atof(nodataopt-answer) for CELL inputs. Reading as int (or
casting to it) will interpret 0x8000 as -2^31, while reading as FP
will interpret it as 2^31.
OTOH, reading as FP allows you to specify +2^31 as the nodata value
when exporting CELL as UInt32. Maybe it would be better to read as
double then cast to the destination type.
Currently, all default nodata values are
within the range of the selected GDAL data type. User-defined nodata
values are tested whether they are within the range of the selected data
type, if not, r.out.gdal in trunk and devbr6 exits with an error
message. r.out.gdal also exits with an error message if the nodata value
is present in the raster to be exported in order to avoid data loss.
This is a rather strict and conservative approach because of the
reported problems with nodata in r.out.gdal exports.
Right.
Even so, 0x8000 is still the ideal value for exporting CELL data
to either UInt32 or Int32. You just have to pick the correct
interpretation of it (unsigned int or signed int respectively).
--
Glynn Clements gl...@gclements.plus.com
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
This is now a mix of r.in.gdal and r.out.gdal. The two modules
complement each other, and I guess a minimum requirement would be that
something exported with r.out.gdal and then imported again with
r.in.gdal should be identical (taking into consideration the region
settings during export) and not have any data loss, be it due to
clipping data or due to nodata problems.
Glynn Clements wrote:
The only fool-proof nodata value for CELL is 0x8000; all other
values may be actual data values.
For grass rasters of type CELL, not for gdal export as Int32. gdalinfo
says NoData Value=2147483648, but cells that should be nodata are
-2147483648 and are thus not recognized as nodata.
It worked as of a few days ago, in the version I have checked out:
raster/r.in.gdal/main.c:780:
if (((GInt32 *) cell)[indx] == (GInt32) dfNoData) {
(GInt32)2147483648 and (GInt32)-2147483648 are equal.
I think the problem is that gdal allows the nodata value to be out of
range with regard to the raster type, e.g. in gdal, a nodata value of
- with GDT_Byte is legal and can be used to effectively disable
nodata information. No idea if this conforms to GeoTIFF specs (probably
yes), but I'm pretty sure that this is the current policy of gdal
(possible with e.g. gdal_translate).
That would also mean that r.out.gdal with type = GDT_Int32 and nodata =
2147483648, NULL cells become -2147483648 but the nodata value in the
metadata stays 2147483648 (gdalinfo on the export output), which in turn
means that other software, also QGIS, does not see any nodata cells in
the export output -information loss about NULL cells. That's why I
wouldn't use 0x8000 as default nodata value for GDT_Int32 in
r.out.gdal even if r.in.gdal has no problems with it.
If it's been changed, use nodata=-0x8000.
That's currently done in r.out.gdal in trunk and devbr6.
Or cast the nodata value
to the source data type then back to double.
For r.out.gdal, there was discussion about not to override user options
and instead issue an error or a warning (going for error now) that the
nodata value is out of range. Currently, all default nodata values are
within the range of the selected GDAL data type. User-defined nodata
values are tested whether they are within the range of the selected data
type, if not, r.out.gdal in trunk and devbr6 exits with an error
message. r.out.gdal also exits with an error message if the nodata value
is present in the raster to be exported in order to avoid data loss.
This is a rather strict and conservative approach because of the
reported problems with nodata in r.out.gdal exports.
Markus M
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
On 17/04/09 16:53, Markus Metz wrote: For r.out.gdal, there was discussion about not to override user options and instead issue an error or a warning (going for error now) that the nodata value is out of range. Currently, all default nodata values are within the range of the selected GDAL data type. User-defined nodata values are tested whether they are within the range of the selected data type, if not, r.out.gdal in trunk and devbr6 exits with an error message. r.out.gdal also exits with an error message if the nodata value is present in the raster to be exported in order to avoid data loss. This is a rather strict and conservative approach because of the reported problems with nodata in r.out.gdal exports. In the current state is there a possibility of using gdal's acceptance of unvalid nodata ? And can we force a nodata value which exists in the map ? I would agree with Dylan that this kind of brute-force method should remain possible, be it through the suggested -f flag. I'm always wary of programs that assume they know better than you what you want ;-) Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Glynn Clements wrote: Dylan Beaudette wrote: 3. Allow the user to specify what they would like NULL cells encoded as. Unless I am overlooking something, it would seem reasonable to export a CELL map to a signed integer format, and use some obvious negative value for NULL: If you're exporting to a 32-bit (signed or unsigned) integer format, the obvious choice for null is 0x8000. That's what GRASS uses internally, so it can never occur as a legitimate value (if a computation ends up producing that value, it will get treated as a null in all regards). 0x8000 as nodata value seems to work with GeoTIFF for Int32 but not for UInt32, here gdalinfo tells me NoData Value=2147483648 which is well within the range of UInt32. It may be a problem that nodataval in r.out.gdal is of type double, so 0x8000 is cast to double and then either back to int or left as double. In the case of UInt32, the raster to be exported is currently read as DCELL, written as GDT_Float64 and then converted to GDT_UInt32. There is a lot of type casting going on in r.out.gdal, making tests a bit complicated. In the end, the value assigned to NULL cells must match the nodata value in the metadata, and a safe default choice seems to be type min or type max. NaN seems to work for Float32 and Float64, at least gdal reports NoData Value=nan and QGIS display is ok for GTiff, HFA, PAux, EHdr. NaN is constructed with 0.0/0.0, idea from r.univar. IMHO, using 0x8000 or similar can be dangerous (not sure about NaN) because I don't know how these values are supported by all the different file formats and all the different applications. Testing welcome! I would suggest to use some fool-proof default nodata value for GDAL integer types, i.e. type min or type max, check if it works, otherwise abort and ask the user to specify a nodata value. For GDAL floating point types, it could be recommended to use the default nodata value (NaN) instead of specifying a nodata value. Markus M ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
On 15/04/09 19:22, Dylan Beaudette wrote: On Wednesday 15 April 2009, GRASS GIS wrote: #73: r.out.gdal tiff output does not work --+ - Reporter: helena | Owner: grass-dev@lists.osgeo.org Type: defect | Status: new Priority: critical | Milestone: 6.4.0 Component: Raster | Version: svn-trunk Resolution: |Keywords: r.out.gdal, tiff Platform: Unspecified | Cpu: Unspecified --+ - Comment (by hamish): Replying to [comment:32 mmetz]: My patch only adds safety checks to make sure that the selected GDAL data type covers the range of the raster band to be exported and that the given null value is within the range of the selected GDAL data type. That would be an enhancement, not a bugfix. Pretend you are r.out.gdal iterating through the raster cells to output. You come across a NULL cell. You have to write down some value, and are completely constrained to options available in the output type. What do you write out? Overflow/wrap the type? type min? type max? well (as things were) whatever you do is going to be arbirtary, and thus could lead to data loss, which is grounds for a most serious blocker bug. I haven't thought a great deal about this, but from what I have been reading and recalling past messages on the GDAL mailing list-- there just aren't any general approaches to preserving NULL values in a geotiff when you need the entire range of a given datatype to represent your information. I think that several things can be done to address this bug, some of which have been mentioned: 1. if there are any NULL cells, warn the user- possibly refuse to export the image in the current format (BYTE, etc.) 1a. if the selected output format cannot contain the entire range, or precision, of input map data (input is FCELL, output is BYTE) give a warning or fail with error message. 2. include a 'force' option to make r.out.gdal do exactly what your command line instructions suggest-- possibly leading to corruption of NULL data 3. Allow the user to specify what they would like NULL cells encoded as. Unless I am overlooking something, it would seem reasonable to export a CELL map to a signed integer format, and use some obvious negative value for NULL: # assuming that all REAL data is within 0-255 r.out.gdal in=CELLMAP out=map.tif format=SIGNEDINT null=-1 also, because the manual has been updated. that is easily modified, so shouldn't stop us. In any case, it would be great if this ticket could be solved before the final release of 6.4. yup I would like to see the above changes, however I cannot actually implement them. Thanks to everyone who has been working on this important module. AFAIK, except for number 1, all are implemented already. Currently there is an error if the null value does not fit into the data range of the selected output file type, but not if the selected value for null data is present in the data as a valid value. But as Markus explained, checking for this would be quite intense, so IMHO a warning that there are null values and that there is the possibility of loosing information if the user does not watch out should be enough. Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
On Wednesday 15 April 2009, Moritz Lennert wrote: On 15/04/09 19:22, Dylan Beaudette wrote: On Wednesday 15 April 2009, GRASS GIS wrote: #73: r.out.gdal tiff output does not work --+- --- - Reporter: helena | Owner: grass-dev@lists.osgeo.org Type: defect | Status: new Priority: critical | Milestone: 6.4.0 Component: Raster | Version: svn-trunk Resolution: |Keywords: r.out.gdal, tiff Platform: Unspecified | Cpu: Unspecified --+- --- - Comment (by hamish): Replying to [comment:32 mmetz]: My patch only adds safety checks to make sure that the selected GDAL data type covers the range of the raster band to be exported and that the given null value is within the range of the selected GDAL data type. That would be an enhancement, not a bugfix. Pretend you are r.out.gdal iterating through the raster cells to output. You come across a NULL cell. You have to write down some value, and are completely constrained to options available in the output type. What do you write out? Overflow/wrap the type? type min? type max? well (as things were) whatever you do is going to be arbirtary, and thus could lead to data loss, which is grounds for a most serious blocker bug. I haven't thought a great deal about this, but from what I have been reading and recalling past messages on the GDAL mailing list-- there just aren't any general approaches to preserving NULL values in a geotiff when you need the entire range of a given datatype to represent your information. I think that several things can be done to address this bug, some of which have been mentioned: 1. if there are any NULL cells, warn the user- possibly refuse to export the image in the current format (BYTE, etc.) 1a. if the selected output format cannot contain the entire range, or precision, of input map data (input is FCELL, output is BYTE) give a warning or fail with error message. 2. include a 'force' option to make r.out.gdal do exactly what your command line instructions suggest-- possibly leading to corruption of NULL data 3. Allow the user to specify what they would like NULL cells encoded as. Unless I am overlooking something, it would seem reasonable to export a CELL map to a signed integer format, and use some obvious negative value for NULL: # assuming that all REAL data is within 0-255 r.out.gdal in=CELLMAP out=map.tif format=SIGNEDINT null=-1 also, because the manual has been updated. that is easily modified, so shouldn't stop us. In any case, it would be great if this ticket could be solved before the final release of 6.4. yup I would like to see the above changes, however I cannot actually implement them. Thanks to everyone who has been working on this important module. AFAIK, except for number 1, all are implemented already. Currently there is an error if the null value does not fit into the data range of the selected output file type, but not if the selected value for null data is present in the data as a valid value. But as Markus explained, checking for this would be quite intense, so IMHO a warning that there are null values and that there is the possibility of loosing information if the user does not watch out should be enough. Moritz Thanks for the notice on 2-3. Your approach sounds good to me. Cheers, Dylan -- Dylan Beaudette Soil Resource Laboratory http://casoilresource.lawr.ucdavis.edu/ University of California at Davis 530.754.7341 ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Moritz Lennert wrote: On 15/04/09 19:22, Dylan Beaudette wrote: On Wednesday 15 April 2009, GRASS GIS wrote: #73: r.out.gdal tiff output does not work --+ - Reporter: helena | Owner: grass-dev@lists.osgeo.org Type: defect | Status: new Priority: critical | Milestone: 6.4.0 Component: Raster | Version: svn-trunk Resolution: |Keywords: r.out.gdal, tiff Platform: Unspecified | Cpu: Unspecified --+ - Comment (by hamish): Replying to [comment:32 mmetz]: My patch only adds safety checks to make sure that the selected GDAL data type covers the range of the raster band to be exported and that the given null value is within the range of the selected GDAL data type. That would be an enhancement, not a bugfix. Pretend you are r.out.gdal iterating through the raster cells to output. You come across a NULL cell. You have to write down some value, and are completely constrained to options available in the output type. What do you write out? Overflow/wrap the type? type min? type max? well (as things were) whatever you do is going to be arbirtary, and thus could lead to data loss, which is grounds for a most serious blocker bug. I haven't thought a great deal about this, but from what I have been reading and recalling past messages on the GDAL mailing list-- there just aren't any general approaches to preserving NULL values in a geotiff when you need the entire range of a given datatype to represent your information. I think that several things can be done to address this bug, some of which have been mentioned: 1. if there are any NULL cells, warn the user- possibly refuse to export the image in the current format (BYTE, etc.) 1a. if the selected output format cannot contain the entire range, or precision, of input map data (input is FCELL, output is BYTE) give a warning or fail with error message. 2. include a 'force' option to make r.out.gdal do exactly what your command line instructions suggest-- possibly leading to corruption of NULL data 3. Allow the user to specify what they would like NULL cells encoded as. Unless I am overlooking something, it would seem reasonable to export a CELL map to a signed integer format, and use some obvious negative value for NULL: # assuming that all REAL data is within 0-255 r.out.gdal in=CELLMAP out=map.tif format=SIGNEDINT null=-1 also, because the manual has been updated. that is easily modified, so shouldn't stop us. In any case, it would be great if this ticket could be solved before the final release of 6.4. yup I would like to see the above changes, however I cannot actually implement them. Thanks to everyone who has been working on this important module. AFAIK, except for number 1, all are implemented already. Currently there is an error if the null value does not fit into the data range of the selected output file type, but not if the selected value for null data is present in the data as a valid value. But as Markus explained, checking for this would be quite intense, so IMHO a warning that there are null values and that there is the possibility of loosing information if the user does not watch out should be enough. 1. warn the user only if no nodata value is given, otherwise (nodata value was given) assume that the user knows what she/he is doing? 1a. only in the patch which is not submitted, and no precision test. 2. suggested but nowhere implemented. why should we have a force option to override all the requested but not yet present safety checks? 3. was always possible I THINK calculating a reasonable nodata value is both time and memory expensive, maybe there is some ingenious way to avoid memory costs, but I'm pretty sure that two passes for each raster band are necessary, i.e. time intensive. r.out.gdal selects a file format, a suitable data type and a default nodata value if none of them were given, but that's more of a guess, and the user should know best what file format, data type and nodata value is appropriate. It gets all a bit more complicated because r.out.gdal can export several raster maps into one single file if supported by the selected file format. Therefore I would prefer some simple tests and abort with a helpful message if something goes wrong instead of doing fancy automated guessing that would make the code much more complex and thus more difficult to maintain. r.out.gdal supports all raster formats supported by gdal, and that's a lot, becoming more with every new version of gdal. IMHO, r.out.gdal should check if input options would cause corrupted output and otherwise refer to the gdal documentation. I'm busy working on an update for r.out.gdal and should be able to submit to trunk tomorrow. Changes are range-check
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
Dylan Beaudette wrote: 3. Allow the user to specify what they would like NULL cells encoded as. Unless I am overlooking something, it would seem reasonable to export a CELL map to a signed integer format, and use some obvious negative value for NULL: If you're exporting to a 32-bit (signed or unsigned) integer format, the obvious choice for null is 0x8000. That's what GRASS uses internally, so it can never occur as a legitimate value (if a computation ends up producing that value, it will get treated as a null in all regards). -- Glynn Clements gl...@gclements.plus.com ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
On Wednesday 15 April 2009, Glynn Clements wrote: Dylan Beaudette wrote: 3. Allow the user to specify what they would like NULL cells encoded as. Unless I am overlooking something, it would seem reasonable to export a CELL map to a signed integer format, and use some obvious negative value for NULL: If you're exporting to a 32-bit (signed or unsigned) integer format, the obvious choice for null is 0x8000. That's what GRASS uses internally, so it can never occur as a legitimate value (if a computation ends up producing that value, it will get treated as a null in all regards). Thanks Glynn. It would be good to know about 'obvious' values for any given datatype. Cheers, Dylan -- Dylan Beaudette Soil Resource Laboratory http://casoilresource.lawr.ucdavis.edu/ University of California at Davis 530.754.7341 ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
On 24/10/08 14:10, Markus Metz wrote: GRASS GIS wrote: #73: r.out.gdal tiff output does not work --+- Reporter: helena | Owner: grass-dev@lists.osgeo.org Type: defect | Status: new Priority: critical | Milestone: 6.4.0 Component: Raster | Version: svn-trunk Resolution: |Keywords: r.out.gdal, tiff Platform: Unspecified | Cpu: Unspecified --+- Comment (by neteler): Markus (Metz), what about integrating your fixes from http://markus.metz.giswork.googlepages.com/r.out.gdal.conservative.tar.gz ? Markus Sure, no objections from my side, I'm using this version only. But r.out.gdal is a very important module of GRASS and maybe some more testing is required. Also, the new features may not find approval by all, e.g. I've put in rather restrictive NULL cell handling: the user has to specify a nodata value that falls within the range of the selected datatype (Byte, UInt16, Int16, ...), a nodata value is no longer chosen automatically as in the original version. If a nodata value is not given, but NULL cells are present, r.out.gdal aborts with an error message. My version also no longer uses the current region resolution, instead the current region extends are aligned to the resolution of the raster to be exported to avoid any implicit resampling. And the colortable is only exported on request, and then with a warning message. If these changes are ok with you then integrate the changes, otherwise maybe only some, but not all changes could be integrated. More discussion needed on how to change/improve r.out.gdal? +1 for all of these changes. Let's try with those and then discuss. Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
On Oct 24, 2008, at 9:32 AM, Moritz Lennert wrote: On 24/10/08 14:10, Markus Metz wrote: GRASS GIS wrote: #73: r.out.gdal tiff output does not work -- +- Reporter: helena | Owner: grass-dev@lists.osgeo.org Type: defect | Status: newPriority: critical | Milestone: 6.4.0 Component: Raster | Version: svn-trunk Resolution: |Keywords: r.out.gdal, tiff Platform: Unspecified | Cpu: Unspecified -- +- Comment (by neteler): Markus (Metz), what about integrating your fixes from http://markus.metz.giswork.googlepages.com/ r.out.gdal.conservative.tar.gz ? Markus Sure, no objections from my side, I'm using this version only. But r.out.gdal is a very important module of GRASS and maybe some more testing is required. Also, the new features may not find approval by all, e.g. I've put in rather restrictive NULL cell handling: the user has to specify a nodata value that falls within the range of the selected datatype (Byte, UInt16, Int16, ...), a nodata value is no longer chosen automatically as in the original version. If a nodata value is not given, but NULL cells are present, r.out.gdal aborts with an error message. My version also no longer uses the current region resolution, instead the current region extends are aligned to the resolution of the raster to be exported to avoid any implicit resampling. And the colortable is only exported on request, and then with a warning message. If these changes are ok with you then integrate the changes, otherwise maybe only some, but not all changes could be integrated. More discussion needed on how to change/improve r.out.gdal? +1 for all of these changes. Let's try with those and then discuss. Moritz I think those are useful changes and agree with Moritz that we should try them out. maybe add it to svn as r.out.gdal2 so that we can test and compare with the old one and then replace if there are no problems? It would be a huge help to get this resolved, Helena ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
On Friday 24 October 2008, Helena Mitasova wrote: On Oct 24, 2008, at 9:32 AM, Moritz Lennert wrote: On 24/10/08 14:10, Markus Metz wrote: GRASS GIS wrote: #73: r.out.gdal tiff output does not work -- +- Reporter: helena | Owner: grass-dev@lists.osgeo.org Type: defect | Status: newPriority: critical | Milestone: 6.4.0 Component: Raster | Version: svn-trunk Resolution: |Keywords: r.out.gdal, tiff Platform: Unspecified | Cpu: Unspecified -- +- Comment (by neteler): Markus (Metz), what about integrating your fixes from http://markus.metz.giswork.googlepages.com/ r.out.gdal.conservative.tar.gz ? Markus Sure, no objections from my side, I'm using this version only. But r.out.gdal is a very important module of GRASS and maybe some more testing is required. Also, the new features may not find approval by all, e.g. I've put in rather restrictive NULL cell handling: the user has to specify a nodata value that falls within the range of the selected datatype (Byte, UInt16, Int16, ...), a nodata value is no longer chosen automatically as in the original version. If a nodata value is not given, but NULL cells are present, r.out.gdal aborts with an error message. My version also no longer uses the current region resolution, instead the current region extends are aligned to the resolution of the raster to be exported to avoid any implicit resampling. And the colortable is only exported on request, and then with a warning message. If these changes are ok with you then integrate the changes, otherwise maybe only some, but not all changes could be integrated. More discussion needed on how to change/improve r.out.gdal? +1 for all of these changes. Let's try with those and then discuss. Moritz I think those are useful changes and agree with Moritz that we should try them out. maybe add it to svn as r.out.gdal2 so that we can test and compare with the old one and then replace if there are no problems? It would be a huge help to get this resolved, Helena +1 from me. I once it is in the 64 branch I can do some testing. Dylan ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev -- Dylan Beaudette Soil Resource Laboratory http://casoilresource.lawr.ucdavis.edu/ University of California at Davis 530.754.7341 ___ grass-dev mailing list grass-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not work
GRASS GIS wrote:
#73: r.out.gdal tiff output does not work
--+-
Reporter: helena | Owner: grass-dev@lists.osgeo.org
Type: defect | Status: new
Priority: major| Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: |Keywords: r.out.gdal, tiff
--+-
Comment (by hamish):
(I see the same with 6.2.1's old script version of r.out.gdal)
it works if the values are in the range of 0-255:
{{{
#spearfish
r.out.gdal in=fields out=fields.tif type=Byte
qiv fields.tif # qiv is a quick image viewer
}}}
but not if all values are out of range:
{{{
r.info -r elevation.dem
min=1066
max=1840
r.info -t elevation.dem
datatype=CELL
r.out.gdal in=elevation.dem out=elv.tif type=Byte
gdalinfo -stats elv.tif | grep STATISTIC
STATISTICS_MINIMUM=255
STATISTICS_MAXIMUM=255
STATISTICS_MEAN=255
STATISTICS_STDDEV=0
r.in.gdal in=elv.tif out=elv2
r.info -r elv2
min=255
max=255
}}}
we can try with half the raster in range:
{{{
r.mapcalc elev_0 = elevation.dem - 1066
r.info -r elev_0
min=0
max=774
r.out.gdal in=elev_0 out=elv0.tif type=Byte
gdalinfo -stats elv0.tif | grep STATISTIC
STATISTICS_MINIMUM=0
STATISTICS_MAXIMUM=254
STATISTICS_MEAN=138.68943338231
STATISTICS_STDDEV=56.526296226099
}}}
If I load elv0.tif into an image viewer it comes out as all black. But in
Gimp if you use the eyedropper you can see the index number changes, but
the palette is black for all 256 index entries.
gdalinfo confirms palette info was not written to file:
I have noticed that r.out.gdal only generates the palette if
G_read_colors() returns 1, i.e. if the map has its own colour table.
If G_read_colors() falls back to a default rainbow colour table, you
don't get a palette.
I suggest:
--- raster/r.out.gdal/main.c(revision 30389)
+++ raster/r.out.gdal/main.c(working copy)
@@ -125,7 +125,7 @@
GDALSetRasterColorInterpretation(hBand, GPI_RGB);
CPLPopErrorHandler();
-if( G_read_colors(name, mapset, sGrassColors ) == 1 )
+if( G_read_colors(name, mapset, sGrassColors ) = 0 )
{
int maxcolor, i;
CELL min, max;
G_read_colors() returns -1 on error (e.g. the map doesn't exist), 0 if
a default rainbow colour table was generated, and 1 if the map has its
own colour table.
AFAICT, the UInt16 case actually works, insofar as the debug output
with DEBUG=3 and the output from gdalinfo look just like they do for
an 8-bit TIFF which works fine in various viewers.
I wouldn't rule out the possibility that popular image viewers can't
handle 16-bit indexed-colour images.
TIFF's strong-point is that it supports so many different formats.
It's weak point is also that it supports so many different formats. I
doubt that there is a program in existence which can correctly handle
every possible valid TIFF file.
--
Glynn Clements [EMAIL PROTECTED]
___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev
