Re: [gdal-dev] HDF5 and NetCDF now have VSI support
That is good news, I am glad to see that that capability is now available to more people. On Sun, Aug 26, 2018 at 3:07 PM Even Rouault wrote: > Hi, > > I've pushed a change in master that now enables /vsi support for HDF5 for > all > operating systems, and not just Linux >= 4.3. This uses the HDF5 Virtual > File > Driver mechanism: > https://support.hdfgroup.org/HDF5/doc/TechNotes/VFL.html > which allows to plug a custom low-level I/O layer > > (In theory, as netCDF V4 uses HDF5 underneath, this could be available for > it > too, but the use of HDF5 is hidden by the netCDF API, so this would > require > patching libnetcdf) > > Even > > -- > Spatialys - Geospatial professional services > http://www.spatialys.com > -- "I prayed for freedom for twenty years, but received no answer until I prayed with my legs." -- Frederick Douglass ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] HDF5 and NetCDF now have VSI support
Hi, I've pushed a change in master that now enables /vsi support for HDF5 for all operating systems, and not just Linux >= 4.3. This uses the HDF5 Virtual File Driver mechanism: https://support.hdfgroup.org/HDF5/doc/TechNotes/VFL.html which allows to plug a custom low-level I/O layer (In theory, as netCDF V4 uses HDF5 underneath, this could be available for it too, but the use of HDF5 is hidden by the netCDF API, so this would require patching libnetcdf) Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] HDF5 and NetCDF now have VSI support
Ah yes, my apologies. On Wed, Aug 8, 2018 at 8:30 AM Even Rouault wrote: > > A concrete example: you should probably not setup a > > /vsimem asset from one thread while another thread is reading a uffd > backed > > asset > > Just a small correction: James meant here not to mix uffd with the use of > the > API in cpl_virtualmem.h ( CPLVirtualMemNew() and the like ). > > /vsimem/ itself has nothing to do with cpl_virtualmem.h and so can be used > with uffd without problem. > > Even > > -- > Spatialys - Geospatial professional services > http://www.spatialys.com > -- "I prayed for freedom for twenty years, but received no answer until I prayed with my legs." -- Frederick Douglass ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] HDF5 and NetCDF now have VSI support
> A concrete example: you should probably not setup a > /vsimem asset from one thread while another thread is reading a uffd backed > asset Just a small correction: James meant here not to mix uffd with the use of the API in cpl_virtualmem.h ( CPLVirtualMemNew() and the like ). /vsimem/ itself has nothing to do with cpl_virtualmem.h and so can be used with uffd without problem. Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] HDF5 and NetCDF now have VSI support
Hello, Support for use of /vsi paths to HDF5 and NetCDF files has just been merged into master (please see here https://github.com/OSGeo/gdal/pull/786 ). We believe that this will be helpful for many people because it is now possible to work with remote files of those formats on an on-demand basis rather than being required to download them. For example, with an installation built from current master, is now possible to type gdalinfo '/vsis3/nasanex/NEX-GDDP/BCSD/rcp85/day/atmos/tasmax/r1i1p1/v1.0/tasmax_day_BCSD_rcp85_r1i1p1_inmcm4_2100.nc' and receive the appropriate output, or such a path can be used from within a program. All of the various VSI drivers should work (e.g. /vsicurl, /vsitar, et cetera). There are a few caveats to mention. First, this capability makes use of the userfaultfd (user mode page fault handling) capability in recent Linux kernels so you will need Linux 4.3 or later. Also, this capability is read-only at this time. Also, for NetCDF support, you will need libnetcdf 4.5 or later. Also, because files are mapped into virtual memory, users might want to impose a (soft) limit on the number of virtual memory pages that can be consumed by any given file. That is done by setting the GDAL_UFFD_LIMIT configuration option to some integer value indicating the number of pages. One should be careful when using GDAL_UFFD_LIMIT: because it makes use of custom signal handlers for SIGSEGV and SIGBUS, it is important not to do anything in another thread in the same process that also changes the behaviour of those signals while actively using a dataset backed by the uffd machinery. A concrete example: you should probably not setup a /vsimem asset from one thread while another thread is reading a uffd backed asset (it is, however, perfectly safe to do use /vsimem from one thread and uffd-backed assets from another as long as the /vsimem setup [when the changes to signal handling happen] does not overlap in time with the uffd-backed reads). I would like to personally thank Even for his patience and accommodation during this work and for his contributions to it. Thanks, James McClain -- "I prayed for freedom for twenty years, but received no answer until I prayed with my legs." -- Frederick Douglass ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
