[gdal-dev] GDAL Maintainers Meeting Minutes
Howard Butler, Even Rouault, Dan Baston, Javier Jiminez Shaw, and Sean Gillies held the monthly GDAL Maintainers Meeting on 03/28/2024. The following items were discussed and reported upon: Sponsorship Updates --- * Dynamic Graphic, Inc. sponsored GDAL at Supporter level Discussion Items GDAL 3.8.5 RC1 is expected next week. It is intended to be the final 3.8 maintenance release with 3.9.0 being expected in May 2024 Maintenance activities update -- Dan * Addressed a few Python bindings crash bugs * Python API documentation cleanups and reorganization to make discovery of methods groups by typical usage stories Alessandro * GDAL PR reviews * Vector file with Polygon Z with 2 coords causing errors https://github.com/OSGeo/gdal/issues/9326 * GTiff: endless loop when resampling (bilinear) under some circumstances https://github.com/OSGeo/gdal/issues/9427 *Inconsistent Output Dimensions from gdal.warp() with Identical outputBounds https://github.com/OSGeo/gdal/issues/9467 * Small improvement to code formatting to add line breaks after methods definition https://github.com/OSGeo/gdal/pull/9498 Even * Exploring implementation of a consistent argument parsing library for all of the GDAL and OGR command line utilities. This library-wide update will provide a number of quality-of-life improvements for users of the command line utilities: * Usage messages will be consistently formatted * Longer help messages * Multiple specification of the same arguments should work more consistently and not fail silently in some cases * Potential for leverage of same framework to specify input for driving CLI activity with config files * Ability to remove lots of boilerplate CLI parsing code in the utilities * RFC99 was merged that provided preliminary implementation to address management and bookkeeping of coordinate precision in OGR drivers that can leverage it. See https://github.com/rouault/gdal/blob/rfc99_text/doc/source/development/rfc/rfc99_geometry_coordinate_precision.rst for discussion * Facilitated the release of OpenJPEG 2.5.2. See https://github.com/uclouvain/openjpeg/releases/tag/v2.5.2 for more information * libtiff PSC motion passed and the project is now managed under GDAL-like processes * Added a getDataset method on Layers to ensure that library users are not required to memoize Layer/Dataset relationships * Action on over 100 tickets and PRs The next GDAL Maintainers Meeting is 04/25/2024 at 9:00 EDT. Any PSC members are welcome to join by reaching out to me for an invite. Howard ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] JPEG2000 / GMLJP2: parameters seem to be ignored (offset location)
This might be tricky, because JPEG2000 itself has a concept of a image space origin which is not necessarily (0,0) with the (XOSiz, YOSiz) fields of the SIZ marker, although most of the time it is 0,0. I don't remember how the GDAL JPEG2000 drivers behave regarding that, if they ignore it (that is the GDAL 0,0 pixel is the one at (XOSiz,YOSiz)), of if the first valid pixel in GDAL image space is actually (XOSiz, YOSiz) and you have padding pixels between 0 and XOSiz/YOSiz. You can get the XOSiz, YOSiz value with the https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/dump_jp2.py script Depending on how drivers behave, the combination of this JPEG2000 imaging (XOSiz, YOSiz) concept with the gml:low shift from GMLJP2 might actually cancel and lead to the right result... In any case data producers are calling for troubles by using any of those mechanisms... Le 29/03/2024 à 09:45, Philippe Ghesquiere via gdal-dev a écrit : Hi all, I have some questions about GMLJP2 parameters concerning location. *1) Description *Some JP2K images have a pixel origin different than (0 ; 0) (DIMAP products for example). In such cases, the GML header looks like like: *7 7 <== Different than "0 0"* 1000 1000 This origin coordinate should be taken into account for pixel location (See OGC 05-047r3 p93): Index position of the first grid post, which shall lie somwhere in the GridEnvelope. If absent, the startPoint is equal to the value of gridEnvelope::low from the grid definition. It seems that this offset is not taken into account by GDAL location function. *2) How to reproduce* 2.1) Data creation Reference (no offset) image: gdal_create -of JP2OpenJPEG -co GMLJP2=YES -outsize 100 100 -bands 3 -burn 0 -burn 255 -burn 0 -a_srs epsg:32632 -a_ullr 50 500100 500100 50 Offset_0.jp2 Image with offset: sed s/"0 0"/"7 7"/g Offset_0.jp2 > Offset_7.jp2 2.2) Location test echo "0 0" | gdaltransform Offset_0.jp2 Offset_7.jp2 0 0 0 *3) Question:* Shouldn't we have a 7 pixels offset between the 2 images Cheers Philippe The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] [EXTERNAL] [BULK] Re: Experience with slowness of free() on Windows with lots of allocations?
for tcmalloc do you need master? this recent release seems to have CMake https://github.com/gperftools/gperftools/releases/tag/gperftools-2.15 Of course, I do not mean to force the usage of it. But could be a suggestion in case we do not find anything better and a user has problems. Or a way to inspire later research. For us it is definitely helping. Cheers, Javier On Thu, 21 Mar 2024 at 14:59, Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > I've played with VirtualAlloc(NULL, SINGLE_ALLOC_SIZE, MEM_COMMIT | > MEM_RESERVE, PAGE_READWRITE), and it does avoid the performance issue. > However I see that VitualAlloc() allocates by chunks of 64 kB, so depending > on the size of a block, it might cause significant waste of RAM, so that > can't be used as a direct replacement of malloc(). > > My inclination would be to perhaps have an optional config option like > GDAL_BLOCK_CACHE_USE_PRIVATE_HEAP that could be set, and when doing so it > would use HeapCreate(0, 0, GDAL_CACHEMAX) to create a heap only used by the > block cache. Not ideal, since that would reserve the whole GDAL_CACHEMAX > (but for a large enough processing, you'll end up consuming it), but it has > the advantage of not being extremely intrusive either... and could be > easily ditched/replaced by something better in the future. > > Regarding tcmalloc, I've had to use it on Linux too, but only on scenarios > involving multithreading where it helps reducing RAM fragmentation: cf > https://gdal.org/user/multithreading.html#ram-fragmentation-and-multi-threading > . I've just tried quickly to use it on Windows to test it on the scenario, > but didn't really manage to make it work. Even building it was challenging. > Actually I tried https://github.com/gperftools/gperftools and I had to > build from master since the latest tagged version doesn't build with CMake > on Windows. But then nothing happens when linking tcmalloc_minimal.lib > against my toy app. I probably missed something. > > Anyway I don't really think we can force tcmalloc to be used in GDAL, as a > library. Unless there would be a way to have its allocator to be optionnaly > used at places that we control (ie explicitly call tc_malloc / tc_free), > and not replace the default malloc / free etc, which might be undesirable > when GDAL is just a component of a larger application. > > Disabling entirely the block cache (or setting it to a minimum value) is > only a workable option for uncompressed formats, or if you use per-band > blocks (INTERLEAVE=BAND in GTiff language) and not one block for all bands > (INTERLEAVE=PIXEL), otherwise you'll pay multiple time the decompression. > Le 21/03/2024 à 14:38, Meyer, Jesse R. (GSFC-618.0)[SCIENCE SYSTEMS AND > APPLICATIONS INC] via gdal-dev a écrit : > > +1. We use a variety of hand-rolled VirtualAlloc based (for basic tasks, > a simple pointer bump, and for more elaborate needs, a ‘buddy’) allocators, > some of which try to be smart about memory usage via de-committing > regions. In our work, we tend to disable the GDAL cache entirely and rely > on the file system’s file cache instead, which is a simplification we can > make but is surely untenable in general here. > > > > *From: *gdal-dev > on behalf of Abel Pau via gdal-dev > > *Reply-To: *Abel Pau > *Date: *Thursday, March 21, 2024 at 4:51 AM > *To: *"gdal-dev@lists.osgeo.org" > > *Subject: *[EXTERNAL] [BULK] Re: [gdal-dev] Experience with slowness of > free() on Windows with lots of allocations? > > > > *CAUTION:* This email originated from outside of NASA. Please take care > when clicking links or opening attachments. Use the "Report Message" > button to report suspicious messages to the NASA SOC. > > > > Hi Even, > > > > you’re right. We also know that. When programming the driver I took it in > consideration. Our solution is not rely on windows to make a good job with > memory and we try to reuse as memory as possible instead of use calloc/free > freely. > > > > For instance, in the driver, for each feature I have to get or write the > coordinates. I could do it every time I have to, so lots of times: create > memory for reading, and then put them on the feature, and then free... so > many times. What I do? When opening the layer I create some memory blocs of > 250 Mb (due to the format itself) and I use that created memory to manage > whatever I need. And when closing, I free it. > > > > While doing that I observed that sometimes I have to use GDAL code that > doesn’t take it in consideration (CPLRecode() for instance). Perhaps it > could be improves as well. > > > > Thanks for noticing that. > > > > *De:* gdal-dev > *En nombre de *Javier Jimenez Shaw via > gdal-dev > *Enviado el:* dijous, 21 de març de 2024 8:27 > *Para:* Even Rouault > > *CC:* gdal dev > *Asunto:* Re: [gdal-dev] Experience with slowness of free() on Windows > with lots of allocations? > > > > In my company we confirmed that "Windows heap allocation mechanism sucks." > > Closing the
[gdal-dev] JPEG2000 / GMLJP2: parameters seem to be ignored (offset location)
Hi all, I have some questions about GMLJP2 parameters concerning location. *1) Description*Some JP2K images have a pixel origin different than (0 ; 0) (DIMAP products for example). In such cases, the GML header looks like like: *7 7 <== Different than "0 0"* 1000 1000 This origin coordinate should be taken into account for pixel location (See OGC 05-047r3 p93): Index position of the first grid post, which shall lie somwhere in the GridEnvelope. If absent, the startPoint is equal to the value of gridEnvelope::low from the grid definition. It seems that this offset is not taken into account by GDAL location function. *2) How to reproduce* 2.1) Data creation Reference (no offset) image: gdal_create -of JP2OpenJPEG -co GMLJP2=YES -outsize 100 100 -bands 3 -burn 0 -burn 255 -burn 0 -a_srs epsg:32632 -a_ullr 50 500100 500100 50 Offset_0.jp2 Image with offset: sed s/"0 0"/"7 7"/g Offset_0.jp2 > Offset_7.jp2 2.2) Location test echo "0 0" | gdaltransform Offset_0.jp2 Offset_7.jp2 0 0 0 *3) Question:* Shouldn't we have a 7 pixels offset between the 2 images Cheers Philippe The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
