Re: [gdal-dev] gdalwarp integer overflow
On 12/03/2013 04:56 PM, Even Rouault wrote:
Hi,
do you have many bands and what is their data type ? Because the check for
integer overflows does nSrcXSize * nSrcYSize * nWordSize * nBandCount.
nSrcXSize * nSrcYSize by itself is 214 659 158. So it means that you have more
than 10 bands of Byte type, or other combinations of band number/data type that
cause the number to go over 2 billion.
I think the cause of the issue is the large over sampling you are attempting.
This requires to fetch an anourmous source area for a tinish target area. I
suggest you do gdalwarp in several times by using decreasing sizes. Perhaps just
an intermediate state will be necessary.
In the situation, the warping algorithm could also be smarter and realize that
in nearest neighbour resampling it doesn't need to fetch a source area but just
fetch individual source pixels. But that's a pretty unusual case (usually the
number of source pixels more or less matches the number of target pixels)
Even
Even, thank you for this clear explanation. I'll try the 2-steps workaround.
Vincent.
Hi,
I get the following error (integer overflow) when running gdalwarp on a
very large vrt to reproject to a very small tif:
gdalwarp -overwrite -t_srs epsg:4326 -te -180 -90 180 90 -ts 36 18 -r
near ${YEAR}/world_${YEAR}_sinu.vrt ${YEAR}/world_${YEAR}_ll_overview.tif
Creating output file that is 36P x 18L.
Processing input file 2013/world_2013_sinu.vrt.
Using internal nodata values (e.g. 0) for image 2013/world_2013_sinu.vrt.
Copying nodata values from source 2013/world_2013_sinu.vrt to
destination 2013/world_2013_ll_overview.tif.
0...10...20...30...40...50...60ERROR 1: Integer overflow :
nSrcXSize=22358, nSrcYSize=9601
I also tried -r average, which gives the same error.
Note that this might be a less than average usecase: I am running
gdalwarp on a vrt (built with gdalbuildvrt) of dimensions 172800, 33600
and sinusoidal projection, to create a tif with dimensions of 36, 18 (10
deg/pixel) and latlon wgs84 projection. This because ultimately I want
to create latlon tiles of 10x10 degrees, but want to avoid creating
empty tiles, so I'll use gdallocationinfo on the 36x18 tif to check if
this 10x10 degree square actually has data.
Any insights or tips on what could be wrong or how to debug/prevent this
are welcome (e.g. should I try some specific warpoptions?)
Thanks in advance,
Vincent.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Setting the noData flag in a GeoTIFF
Hi Jukka, Good point. In fact, that's one of the reasons I want to set the noData flag. I figure when compressing hopefully the compressor will see that it is noData and will treat it differently from the way it would an actual value (which results in things like your mentioned 255,255,254) so I won't get the white-border-effect. I guess I'll find out if/when I get to that point. Cheers, Jonathan On 4 December 2013 06:58, Jukka Rahkonen jukka.rahko...@mmmtike.fi wrote: Jonathan Moules jonathanmoules at warwickshire.gov.uk writes: Hi Even, That worked great. It figures that I'd tried about 6 different permutations, including a few without -hidenodata; but none of them was the right one it seems. It's no problem recreating the tifs as they're small test tifs that takes a few seconds to process. Many thanks!Jonathan Hi, Be warned that the result may not be what you believe. You are compressing to jpeg and as a lossy method it does not store constant pixel values for the whole nodata area. If 255-255-255 pixels are nodata values you will get also pixels like 255-255-254 here and there and they will not be transparent in applications. That may not be a problem for you if you combine rectangular images with full of data into a mosaic but if you warp images one by one into another projection so that they will get rotated and then want to use the warped images side by side you would see that the overlapping areas look bad. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Setting the noData flag in a GeoTIFF
Jonathan Moules jonathanmoules at warwickshire.gov.uk writes: Hi Jukka,Good point. In fact, that's one of the reasons I want to set the noData flag. I figure when compressing hopefully the compressor will see that it is noData and will treat it differently from the way it would an actual value (which results in things like your mentioned 255,255,254) so I won't get the white-border-effect. I guess I'll find out if/when I get to that point. Hi, As far as I know GDAL does not do miracles. You cannot make a perfect result with rotated images, noData value and jpeg compression if you warp first and mosaic afterwards. If you want to use jpeg compression, as I suppose you do, turn the order and mosaic first - warp next. Build a virtual mosaic when you still play in the world of the native projection and warp images to target projection by using –projwin. That way at least the images from the middle parts of your site will be rectangular and without noData areas at all in the target srs and you will have no issues with jpeg compression. You will still get non-transparent pixels into the borders of your site but you can create those images without compression or with lossless LZW/deflate compression. You are anyways so well oriented in doing controlled tests that you will probably do little experiments with your own data and mosaics of size 2x2 or 3x3 tiles before running huge batches. However, if your originals don't have white borders of noData areas withing the image area then you will not have them either after running your commands. gdalbuildvrt -hidenodata -srcnodata 255 -vrtnodata 255 -input_file_list tiff_list.txt 255.vrt gdal_translate -of GTiff -co TILED=YES -co BIGTIFF=YES -co COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co PHOTOMETRIC=YCBCR 255.vrt 255mask.tif In this case using noData is not necessary either and may actually burn holes into your images if there are some extremely light areas of dry, plain terrain or the sun is reflecting from the water bodies and roofs of the buildings etc. -Jukka- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] How to get coordinates of original corners from warped image
On Sat, Nov 30, 2013 at 1:27 PM, Lauri Kajan lauri.ka...@gmail.com wrote: Hi all I have a set of aerial bird's view images and now I need to get extents of those images as polygons. I thought to add GCPs to images and possibly somehow to use the same transformation that is used when warping the images with perspective algorithm, to calculate coordinates for corners of the original images. Get the coordinates from you projected Image then you can use the cs2cs to re-projected those coordinates back to your original CRS Is this possible with python? yes, you can use the pyproj lib Thanks -Lauri ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] VS2010-VS2013 s57
Hi,
I have such error: the GDAL compiled with VS2010-VS2013 in s57 driver
loose all additional fields values. But the same code compiled with gcc
or previous VS works fine.
I found the root of problems here (ogr\ogrsf_frmts\s57\s57reader.cpp:932):
const char *pszAcronym = poRegistrar-GetAttrAcronym(nAttrId);
iField = poFeature-GetDefnRef()-GetFieldIndex(pszAcronym);
The pszAcronym always empty.
The problem comes from this function (ogr\ogrsf_frmts\s57\s57.h:140):
const char *GetAttrAcronym( int i )
{ return GetAttrInfo(i) == NULL ? NULL :
aoAttrInfos[i]-osAcronym; }
It seems to me that during execution this function I have |situation
when||c_str()|result becomes invalid (the|std::string|is destroyed or a
non-const member function of the string is called).
If I change function
const char *GetAttrAcronym( int i )
{ return GetAttrInfo(i) == NULL ? NULL :
aoAttrInfos[i]-osAcronym.c_str(); }
the problem gone.
So I need some confirmation/verification my ideas.
If I'm right, I can make changes to s57 driver.
Best regards,
Dmitry
---
Это сообщение свободно от вирусов и вредоносного ПО благодаря защите от вирусов
avast!
http://www.avast.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] VS2010-VS2013 s57
Dmitriy,
It is not clear to me why this should be necessary. osAcronym is a long
lived std::string (well CPLString derived from std::string) living in the
registrars attribute list. I assume the following method is used to
convert the CPLString to const char * which should amount to the same
thing you did.
operator const char* (void) const { return c_str(); }
Perhaps there is some subtle reason I don't see that the compiler is
creating a temporary std::string in between?
In any event, if you file a ticket I can apply this change upstream. There
are other accessors on the same class that look like they could have
similar issues.
Best regards,
Frank
On Wed, Dec 4, 2013 at 12:11 PM, Dmitriy Baryshnikov
bishop@gmail.comwrote:
Hi,
I have such error: the GDAL compiled with VS2010-VS2013 in s57 driver
loose all additional fields values. But the same code compiled with gcc or
previous VS works fine.
I found the root of problems here (ogr\ogrsf_frmts\s57\s57reader.cpp:932):
const char *pszAcronym = poRegistrar-GetAttrAcronym(nAttrId);
iField = poFeature-GetDefnRef()-GetFieldIndex(pszAcronym);
The pszAcronym always empty.
The problem comes from this function (ogr\ogrsf_frmts\s57\s57.h:140):
const char *GetAttrAcronym( int i )
{ return GetAttrInfo(i) == NULL ? NULL : aoAttrInfos[i]-osAcronym; }
It seems to me that during execution this function I have situation when
c_str() result becomes invalid (the std::string is destroyed or a
non-const member function of the string is called).
If I change function
const char *GetAttrAcronym( int i )
{ return GetAttrInfo(i) == NULL ? NULL :
aoAttrInfos[i]-osAcronym.c_str(); }
the problem gone.
So I need some confirmation/verification my ideas.
If I'm right, I can make changes to s57 driver.
Best regards,
Dmitry
--
http://www.avast.com/
Это сообщение свободно от вирусов и вредоносного ПО благодаря avast!
Antivirus http://www.avast.com/ защита активна.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
--
---+--
I set the clouds in motion - turn up | Frank Warmerdam,
warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Software Developer
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] VS2010-VS2013 s57
Hi Frank,
You quite right, what
It is not clear to me why this should be necessary. osAcronym is a
long lived std::string (well CPLString derived from std::string)
living in the registrars attribute list. I assume the following
method is used to convert the CPLString to const char * which
should amount to the same thing you did.
but this is not helps, as c_str() method of std::string not return
pointer, but construct new zero ended const char* managed by std::string
(maybe this is specific to VS2010-VS2013). This const char* cleared
after some call to std::string or CPLString and we got empty (cleared)
const char *pszAcronym. I think the VS2010-VS2013 compiler do this in
getter by index, operator const char* or the whole GetAttrInfo(i) ==
NULL ? NULL : aoAttrInfos[i]-osAcronym (maybe MS change the order
parameter calculation or optimization). Maybe const (operator const
char* (void) const) in non const (const char *GetAttrAcronym( int i ))
function VS2010-VS2013 specific behaviour.
OK, I fire the ticket. As I have committer rights, my I add changes,
corresponding to the ticket?
Best regards,
Dmitry
05.12.2013 0:19, Frank Warmerdam пишет:
Dmitriy,
It is not clear to me why this should be necessary. osAcronym is a
long lived std::string (well CPLString derived from std::string)
living in the registrars attribute list. I assume the following
method is used to convert the CPLString to const char * which should
amount to the same thing you did.
operator const char* (void) const { return c_str(); }
Perhaps there is some subtle reason I don't see that the compiler is
creating a temporary std::string in between?
In any event, if you file a ticket I can apply this change upstream.
There are other accessors on the same class that look like they could
have similar issues.
Best regards,
Frank
On Wed, Dec 4, 2013 at 12:11 PM, Dmitriy Baryshnikov
bishop@gmail.com mailto:bishop@gmail.com wrote:
Hi,
I have such error: the GDAL compiled with VS2010-VS2013 in s57
driver loose all additional fields values. But the same code
compiled with gcc or previous VS works fine.
I found the root of problems here
(ogr\ogrsf_frmts\s57\s57reader.cpp:932):
const char *pszAcronym = poRegistrar-GetAttrAcronym(nAttrId);
iField = poFeature-GetDefnRef()-GetFieldIndex(pszAcronym);
The pszAcronym always empty.
The problem comes from this function (ogr\ogrsf_frmts\s57\s57.h:140):
const char *GetAttrAcronym( int i )
{ return GetAttrInfo(i) == NULL ? NULL :
aoAttrInfos[i]-osAcronym; }
It seems to me that during execution this function I have
|situation when||c_str()|result becomes invalid
(the|std::string|is destroyed or a non-const member function of
the string is called).
If I change function
const char *GetAttrAcronym( int i )
{ return GetAttrInfo(i) == NULL ? NULL :
aoAttrInfos[i]-osAcronym.c_str(); }
the problem gone.
So I need some confirmation/verification my ideas.
If I'm right, I can make changes to s57 driver.
Best regards,
Dmitry
http://www.avast.com/
Это сообщение свободно от вирусов и вредоносного ПО благодаря
avast! Antivirus http://www.avast.com/ защита активна.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org mailto:gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
--
---+--
I set the clouds in motion - turn up | Frank Warmerdam,
warmer...@pobox.com mailto:warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
http://pobox.com/%7Ewarmerdam
and watch the world go round - Rush| Geospatial Software Developer
---
Это сообщение свободно от вирусов и вредоносного ПО благодаря защите от вирусов
avast!
http://www.avast.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] VS2010-VS2013 s57
Le mercredi 04 décembre 2013 22:15:58, Dmitriy Baryshnikov a écrit :
Hi Frank,
You quite right, what
It is not clear to me why this should be necessary. osAcronym is a
long lived std::string (well CPLString derived from std::string)
living in the registrars attribute list. I assume the following
method is used to convert the CPLString to const char * which
should amount to the same thing you did.
but this is not helps, as c_str() method of std::string not return
pointer, but construct new zero ended const char* managed by std::string
(maybe this is specific to VS2010-VS2013). This const char* cleared
after some call to std::string or CPLString and we got empty (cleared)
const char *pszAcronym. I think the VS2010-VS2013 compiler do this in
getter by index, operator const char* or the whole GetAttrInfo(i) ==
NULL ? NULL : aoAttrInfos[i]-osAcronym (maybe MS change the order
parameter calculation or optimization). Maybe const (operator const
char* (void) const) in non const (const char *GetAttrAcronym( int i ))
function VS2010-VS2013 specific behaviour.
OK, I fire the ticket. As I have committer rights, my I add changes,
corresponding to the ticket?
Yes, Dmitriy, please commit your change (GetAttrName() would need the same
change also). Hopefully, it will solve the failure on ogr_s57_6 that have been
seen in http://www.gisinternals.com/sdk/build-output/vc10-20131204-5-52-50-84-
vc10-dev.txt since a few weeks and I couldn't understand.
And actually this must be related to one of my latest change in the S57 driver
where I've restructured that area a bit to gain thread-safety.
I'm not sure if VS2010 is right or wrong in doing what it does, but if the
.c_str() fix the issue, that's always good to have !
Even
Best regards,
Dmitry
05.12.2013 0:19, Frank Warmerdam пишет:
Dmitriy,
It is not clear to me why this should be necessary. osAcronym is a
long lived std::string (well CPLString derived from std::string)
living in the registrars attribute list. I assume the following
method is used to convert the CPLString to const char * which should
amount to the same thing you did.
operator const char* (void) const { return c_str(); }
Perhaps there is some subtle reason I don't see that the compiler is
creating a temporary std::string in between?
In any event, if you file a ticket I can apply this change upstream.
There are other accessors on the same class that look like they could
have similar issues.
Best regards,
Frank
On Wed, Dec 4, 2013 at 12:11 PM, Dmitriy Baryshnikov
bishop@gmail.com mailto:bishop@gmail.com wrote:
Hi,
I have such error: the GDAL compiled with VS2010-VS2013 in s57
driver loose all additional fields values. But the same code
compiled with gcc or previous VS works fine.
I found the root of problems here
(ogr\ogrsf_frmts\s57\s57reader.cpp:932):
const char *pszAcronym = poRegistrar-GetAttrAcronym(nAttrId);
iField = poFeature-GetDefnRef()-GetFieldIndex(pszAcronym);
The pszAcronym always empty.
The problem comes from this function (ogr\ogrsf_frmts\s57\s57.h:140):
const char *GetAttrAcronym( int i )
{ return GetAttrInfo(i) == NULL ? NULL :
aoAttrInfos[i]-osAcronym; }
It seems to me that during execution this function I have
|situation when||c_str()|result becomes invalid
(the|std::string|is destroyed or a non-const member function of
the string is called).
If I change function
const char *GetAttrAcronym( int i )
{ return GetAttrInfo(i) == NULL ? NULL :
aoAttrInfos[i]-osAcronym.c_str(); }
the problem gone.
So I need some confirmation/verification my ideas.
If I'm right, I can make changes to s57 driver.
Best regards,
Dmitry
-
--- http://www.avast.com/
Это сообщение свободно от вирусов и вредоносного ПО благодаря
avast! Antivirus http://www.avast.com/ защита активна.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org mailto:gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
---
Это сообщение свободно от вирусов и вредоносного ПО благодаря защите от
вирусов avast! http://www.avast.com
--
Geospatial professional services
http://even.rouault.free.fr/services.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
