Lucena, Ivan wrote:
Hi there,
I am running a Python script that goes through a relatively large number of
single band raster files (320) and aggregates it in a big-geotiff (around
7Gb) and I am facing three basic problems *poor performance*, *wrong
results*" and *lost of metadata*.
I need that geotiff to be interleaved by pixel since the goal is to query
for historical values in a given location and return then in the best
response time possible.
So I am using this create options:
...
I also need to add custom metadata entries to the dataset and to the
individuals bands so that we know their data stamps.
That is how I loop to all the files and copy then to the output big-geotiff
file:
> --
> gdal.SetCacheMax(500)
> count = 1
> for i in list_tsf:
> input_dst = gdal.Open( f )
> input_band = input_dst.GetRasterBand( 1 )
> output_band = output_dst.GetRasterBand( count )
> output_band.WriteRaster( 0, 0, x_size, y_size,
input_band.ReadRaster( 0, 0, x_size, y_size ) )
> output_band.FlushCache()
> count = count + 1
> --
*Performance*: The process takes more than 6 hours.
Ivan,
I'm not surprised that the performance sucks. Doing
random reads and writes to a pixel interleaved file is
triggering a great deal of extra data movement.
> (Wrong Results*: All the pixels are zero (0)
If I just change the script on 'INTERLEAVE=PIXEL' to 'INTERLEAVE=BAND' the
whole process takes around 30 minutes.
OK, it sounds like the pixels all being zero is a bug, and
it would be good to file a ticket demonstrating this problem.
Hopefully a somewhat minimalist example of the problem.
I can then use gdal_translate to convert from GTIFF to GTIFF with -co
'INTERLEAVE=PIXEL'. That takes another 30 minutes.
*Lost Metadata*: That looks like a solution except that I will then loose
all my custom metadata entries.
Are your custom metadata values in a non-default
domain? If so they will not be copied by default,
and you would need to add something in the script
to copy them after gdal_translate (or driver.CreateCopy()).
Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, [email protected]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent
_______________________________________________
gdal-dev mailing list
[email protected]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
