Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading
Hi, Motion passed with +1 from PSC members KurtS, HowardB, JukkaR, JavierJS and me Even Le 15/11/2023 à 10:51, Even Rouault via gdal-dev a écrit : Hi, Motion: adopt RFC 96: Deferred C++ plugin loading https://github.com/OSGeo/gdal/pull/8648 Pre-rendered view: https://github.com/rouault/gdal/blob/rfc96_text/doc/source/development/rfc/rfc96_deferred_plugin_loading.rst Starting with my +1, Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading
Hi Sean On Thu, 16 Nov 2023 at 12:10, Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > > > > I think this makes great sense for the project. I don't yet understand > > what it means for an enterprise like Rasterio's PyPI wheels. > > I'd say it probably changes nothing. The RFC just postpones the time > where the plugins are loaded, but the fact that they are dlopen()'ed > (early or late) probably makes them non discoverable by delocate, since > libgdal doesn't link to them in a way that is advertised in its shared > library metadata. If your plan is to still have a rasterio wheel with a > monolithic GDAL, then you don't need to build GDAL drivers as plugins > and this RFC doesn't change anything to the status quo. > > I'm not familiar at all with the wheel Python packaging tools, but if > you'd want to have GDAL plugins in separate package(s) then you'd need > to have a way of having the gdal_XXX.so / ogr_XXX.so be put in some > known location that can be advertized to libgdal core with > GDAL_DRIVER_PATH. > PyInstaller has a set of hooks for this, where custom behaviour can be scripted for packages that need it. AFAICS auditwheel & delocate don't, and also don't have `--include=my.so` options to override it and "force" libraries to be processed — the same issue crops up for stuff using ctypes IIUC. My horrible workaround idea: building a shim _gdalplugins python module which does nothing except link to all the plugin libraries — that would allow them all to be processed by delocate/auditwheel alongside the real libgdal — so you get libgdal, and all the driver plugin libraries, and their dependencies. Since _gdalplugins is never used/loaded (ideally it'd be dropped) gdal would work via runtime plugin loading. The library renaming that auditwheel/delocate do might get in the way, but hopefully there's a path through that. Cheers, Rob :) ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading
Sounds good to me, Even. Rasterio's wheels can remain at the forefront of terrible for now. On Thu, Nov 16, 2023 at 5:10 AM Even Rouault wrote: > Hi Sean, > > > > I think this makes great sense for the project. I don't yet understand > > what it means for an enterprise like Rasterio's PyPI wheels. > > I'd say it probably changes nothing. The RFC just postpones the time > where the plugins are loaded, but the fact that they are dlopen()'ed > (early or late) probably makes them non discoverable by delocate, since > libgdal doesn't link to them in a way that is advertised in its shared > library metadata. If your plan is to still have a rasterio wheel with a > monolithic GDAL, then you don't need to build GDAL drivers as plugins > and this RFC doesn't change anything to the status quo. > > I'm not familiar at all with the wheel Python packaging tools, but if > you'd want to have GDAL plugins in separate package(s) then you'd need > to have a way of having the gdal_XXX.so / ogr_XXX.so be put in some > known location that can be advertized to libgdal core with > GDAL_DRIVER_PATH. > > Even > > -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading
Hi Sean, I think this makes great sense for the project. I don't yet understand what it means for an enterprise like Rasterio's PyPI wheels. I'd say it probably changes nothing. The RFC just postpones the time where the plugins are loaded, but the fact that they are dlopen()'ed (early or late) probably makes them non discoverable by delocate, since libgdal doesn't link to them in a way that is advertised in its shared library metadata. If your plan is to still have a rasterio wheel with a monolithic GDAL, then you don't need to build GDAL drivers as plugins and this RFC doesn't change anything to the status quo. I'm not familiar at all with the wheel Python packaging tools, but if you'd want to have GDAL plugins in separate package(s) then you'd need to have a way of having the gdal_XXX.so / ogr_XXX.so be put in some known location that can be advertized to libgdal core with GDAL_DRIVER_PATH. Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading
Hi Even, I think this makes great sense for the project. I don't yet understand what it means for an enterprise like Rasterio's PyPI wheels. Here's a refresher for people who aren't familiar with Python packaging tools like delocate, auditwheel, and delvewheel. Today, these tools detect when Rasterio extension modules link libgdal, and then find the libraries that libgdal links via metadata in the shared libraries. This allows libgdal and its dependencies to be relocated so that they can be distributed alongside Rasterio's Python and extension modules. I don't know that delocate et al will be able to find libraries that are dlopened by the new GDAL proxy drivers. Does anyone else know? I'm starting a new job this week and may not have time to experiment until next week. On Wed, Nov 15, 2023 at 2:52 AM Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hi, > > Motion: adopt RFC 96: Deferred C++ plugin loading > > https://github.com/OSGeo/gdal/pull/8648 > > Pre-rendered view: > > > https://github.com/rouault/gdal/blob/rfc96_text/doc/source/development/rfc/rfc96_deferred_plugin_loading.rst > > > Starting with my +1, > > Even > -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading
+1 Javier On Wed, 15 Nov 2023 at 21:44, Rahkonen Jukka via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > +1 > > -Jukka Rahkonen- > > -Alkuperäinen viesti- > Lähettäjä: gdal-dev Puolesta Even > Rouault via gdal-dev > Lähetetty: keskiviikko 15. marraskuuta 2023 11.52 > Vastaanottaja: gdal-dev@lists.osgeo.org > Aihe: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading > > Hi, > > Motion: adopt RFC 96: Deferred C++ plugin loading > ... > Starting with my +1, > > Even > > -- > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading
+1 -Jukka Rahkonen- -Alkuperäinen viesti- Lähettäjä: gdal-dev Puolesta Even Rouault via gdal-dev Lähetetty: keskiviikko 15. marraskuuta 2023 11.52 Vastaanottaja: gdal-dev@lists.osgeo.org Aihe: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading Hi, Motion: adopt RFC 96: Deferred C++ plugin loading ... Starting with my +1, Even -- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading
+1 Howard I'm excited for this to start percolating. Thank you for all of the effort to design and implement it. > On Nov 15, 2023, at 3:51 AM, Even Rouault via gdal-dev > wrote: > > Hi, > > Motion: adopt RFC 96: Deferred C++ plugin loading > > https://github.com/OSGeo/gdal/pull/8648 > > Pre-rendered view: > > https://github.com/rouault/gdal/blob/rfc96_text/doc/source/development/rfc/rfc96_deferred_plugin_loading.rst > > > Starting with my +1, > > Even > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Motion: adopt RFC 96: Deferred C++ plugin loading
+1 KurtS One minor comment... I use pfnOpen in a few of my local fuzzers, but I build statically without plugins, so I think my use won't be impacted. But even if I am impacted, that's my problem and not a responsibility of the GDAL project as I'm using internal GDAL details. > Another potential issue is that if external code would directly use the pfnOpen, pfnCreate, pfnCreateCopy, etc. function pointers of a GDALDriver instance, it would see them null before the actual driver is loading, but direct access to those function pointers has never been documented (instead users should use GDALOpen(), GDALCreate(), GDALCreateCopy() etc), and is not expected to be done by code external to libgdal core. On Wed, Nov 15, 2023 at 1:52 AM Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hi, > > Motion: adopt RFC 96: Deferred C++ plugin loading > > https://github.com/OSGeo/gdal/pull/8648 > > Pre-rendered view: > > > https://github.com/rouault/gdal/blob/rfc96_text/doc/source/development/rfc/rfc96_deferred_plugin_loading.rst > > > Starting with my +1, > > Even > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
