Simon,
unfortunately there are a number of places in the degrib library which
aren't thread-safe, and you just spotted that the errSprintf() routine
was one of them. I've queued a fix for that in
https://github.com/OSGeo/gdal/pull/4830. Could you try it ?
Regarding gdal_translate performance, this is related to something I
mentioned recently (not sure to whom), but you'll get much better
performance if you add -co INTERLEAVE=BAND, so that the output GeoTIFF
is written band after band, to match the most performant access pattern
for reading GRIB files.
Even
Le 15/11/2021 à 01:32, Simon Eves a écrit :
We have recently implemented a geo raster importer, and all seems
fine, except that we hit an issue with a particular GRIB2 file from
the NOAA website, where we get an inconsistent crash inside GDAL after
a few hundred scanlines.
We have seen two different crashes inside GDAL, and a third in one of
our code paths, but given that there is a memory corruption, the
latter is perhaps unsurprising.
All crashes report "double free or corruption (fasttop)".
We are multi-threading the reading, but using a OGRDataSource per
thread. The child threads are only calling GetRasterBand(),
GetRasterDataType() and RasterIO() and only one one band at a time.
The GRIB2 file is 103MB with 73 Float64 bands, but only 2345x1597
"pixels".
We tried converting the file to GeoTIFF with gdal_translate (no
options, just in and out) and it took 28 minutes (on a ~2017 i7 quad
4.2), which is surprising, as we have other GRIB2 files (between 2 and
12MB) which convert "instantly". The resulting GeoTIFF is much bigger
(21x) but seems to import reliably, has basically the same schema (as
reported by gdalinfo) and results in the same data when imported into
our system.
We only get the crash occasionally, and have only been able to trap it
in the Debugger a couple of times, with nothing obviously wrong.
Here is a link to the GRIB2 file in question:
https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing
<https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing>
Attached is the most common stack trace.
With a DEBUG build of GDAL, looks like it's crashing trying to do a
realloc() on "buffer" which is NULL, although that is supposedly a
copy of "&errBuffer" at the frame above which seems fine.
Gonna try robustifying that code and see what happens...
This is all with GDAL 3.2.2 on Ubuntu 20.04 LTS with GCC 9.
--
<http://www.omnisci.com/>
Simon Eves
Senior Graphics Engineer, Rendering Group
100 Montgomery St (5th Floor), San Francisco, CA 94104, USA
Email: [email protected] <mailto:[email protected]> | Cell:
+1 (415) 902-1996
_______________________________________________
gdal-dev mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
