Hi Even, Sorry for the slow response. I was out yesterday and this morning. Testing your fix now.
Simon On Mon, Nov 15, 2021 at 4:59 AM Even Rouault <[email protected]> wrote: > Simon, > > unfortunately there are a number of places in the degrib library which > aren't thread-safe, and you just spotted that the errSprintf() routine was > one of them. I've queued a fix for that in > https://github.com/OSGeo/gdal/pull/4830. Could you try it ? > > Regarding gdal_translate performance, this is related to something I > mentioned recently (not sure to whom), but you'll get much better > performance if you add -co INTERLEAVE=BAND, so that the output GeoTIFF is > written band after band, to match the most performant access pattern for > reading GRIB files. > > Even > Le 15/11/2021 à 01:32, Simon Eves a écrit : > > We have recently implemented a geo raster importer, and all seems fine, > except that we hit an issue with a particular GRIB2 file from the NOAA > website, where we get an inconsistent crash inside GDAL after a few hundred > scanlines. > > We have seen two different crashes inside GDAL, and a third in one of our > code paths, but given that there is a memory corruption, the latter is > perhaps unsurprising. > > All crashes report "double free or corruption (fasttop)". > > We are multi-threading the reading, but using a OGRDataSource per thread. > The child threads are only calling GetRasterBand(), GetRasterDataType() and > RasterIO() and only one one band at a time. > > The GRIB2 file is 103MB with 73 Float64 bands, but only 2345x1597 "pixels". > > We tried converting the file to GeoTIFF with gdal_translate (no options, > just in and out) and it took 28 minutes (on a ~2017 i7 quad 4.2), which is > surprising, as we have other GRIB2 files (between 2 and 12MB) which convert > "instantly". The resulting GeoTIFF is much bigger (21x) but seems to import > reliably, has basically the same schema (as reported by gdalinfo) and > results in the same data when imported into our system. > > We only get the crash occasionally, and have only been able to trap it in > the Debugger a couple of times, with nothing obviously wrong. > > Here is a link to the GRIB2 file in question: > > > https://drive.google.com/file/d/12Fo6jnIhxzCvnSsup9n0kHVKy9lrHD2l/view?usp=sharing > > Attached is the most common stack trace. > > With a DEBUG build of GDAL, looks like it's crashing trying to do a > realloc() on "buffer" which is NULL, although that is supposedly a copy of > "&errBuffer" at the frame above which seems fine. > > Gonna try robustifying that code and see what happens... > > This is all with GDAL 3.2.2 on Ubuntu 20.04 LTS with GCC 9. > > -- > <http://www.omnisci.com/> > Simon Eves > Senior Graphics Engineer, Rendering Group > 100 Montgomery St (5th Floor), San Francisco, CA 94104, USA > > > Email: [email protected] | Cell: +1 (415) 902-1996 > > > _______________________________________________ > gdal-dev mailing > [email protected]https://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- http://www.spatialys.com > My software is free, but my time generally not. > > -- <http://www.omnisci.com/> Simon Eves Senior Graphics Engineer, Rendering Group 100 Montgomery St (5th Floor), San Francisco, CA 94104, USA Email: [email protected] | Cell: +1 (415) 902-1996
_______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
