Re: [gdal-dev] Block cache and VRT sources
Thanks, Even. I think I've got my head around it. Since I've never written a driver, I've never fully learned the difference between IReadBlock and IRasterIO or where one is used instead of the other. But I'm learning that now. On Fri, Apr 19, 2024 at 9:35 AM Even Rouault wrote: > > Le 19/04/2024 à 17:28, Sean Gillies a écrit : > > Even, > > Does the shared attribute of a VRT sourceFilename element not affect > caching at all? > > If shared is set to 0, then one GDALDataset per VRTSource will be opened. > This has little benefit. > > The scope of sharing was initially greater since it extented to > potentially several GDALDataset instance of VRT, but this wasn't safe in > multithreaded scenarios where you would read one VRT in a thread, another > one in another thread, but both would point to let's same the same TIFF > GDALDataset. Hence the scope of sharing was reduced to a single GDALDataset > VRT instance to be safe by default. > > Is the cache avoided so that potentially stale data isn't propagated, or > for other reasons? > > Are you reacting to "if it is triggered. VRTSource::IRasterIO() calls > IRasterIO() on the source band, which doesn't necessarily always trigger > block-based reading" ? I just meant that calling IRasterIO() on a > GDALDataset/GDALRasterBand doesn't necessarily cause the block cache to > trigger. This is dependent of the driver and the parameters of the request. > And this isn't specific to use it from a VRT. > > > On Fri, Apr 19, 2024 at 9:09 AM Even Rouault > wrote: > >> Sean, >> >> Within a given GDALDataset opened on a VRT, if the VRT references several >> times the same source, only one GDALDataset will be opened for it, so you >> may benefit from the block cache mechanism (if it is triggered. >> VRTSource::IRasterIO() calls IRasterIO() on the source band, which doesn't >> necessarily always trigger block-based reading). But if you open another >> VRT (or the same one), it will not share the same GDALDataset for sources >> that may be common with the first one, so no re-use of existing block >> cache. For network sources, the I/O cache at the /vsicurl/ level works >> however on filenames, not VSIFILE* instances, so you will save network reads >> >> Even >> Le 19/04/2024 à 16:48, Sean Gillies via gdal-dev a écrit : >> >> Happy Friday, folks! >> >> Are the source rasters of a VRT added to the block cache such that >> different VRTs using the same source can avoid reads from disk or the >> network? Or is it intended that the VSI cache covers this need instead? >> >> Thanks, >> >> -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Block cache and VRT sources
Le 19/04/2024 à 17:28, Sean Gillies a écrit : Even, Does the shared attribute of a VRT sourceFilename element not affect caching at all? If shared is set to 0, then one GDALDataset per VRTSource will be opened. This has little benefit. The scope of sharing was initially greater since it extented to potentially several GDALDataset instance of VRT, but this wasn't safe in multithreaded scenarios where you would read one VRT in a thread, another one in another thread, but both would point to let's same the same TIFF GDALDataset. Hence the scope of sharing was reduced to a single GDALDataset VRT instance to be safe by default. Is the cache avoided so that potentially stale data isn't propagated, or for other reasons? Are you reacting to "if it is triggered. VRTSource::IRasterIO() calls IRasterIO() on the source band, which doesn't necessarily always trigger block-based reading" ? I just meant that calling IRasterIO() on a GDALDataset/GDALRasterBand doesn't necessarily cause the block cache to trigger. This is dependent of the driver and the parameters of the request. And this isn't specific to use it from a VRT. On Fri, Apr 19, 2024 at 9:09 AM Even Rouault wrote: Sean, Within a given GDALDataset opened on a VRT, if the VRT references several times the same source, only one GDALDataset will be opened for it, so you may benefit from the block cache mechanism (if it is triggered. VRTSource::IRasterIO() calls IRasterIO() on the source band, which doesn't necessarily always trigger block-based reading). But if you open another VRT (or the same one), it will not share the same GDALDataset for sources that may be common with the first one, so no re-use of existing block cache. For network sources, the I/O cache at the /vsicurl/ level works however on filenames, not VSIFILE* instances, so you will save network reads Even Le 19/04/2024 à 16:48, Sean Gillies via gdal-dev a écrit : Happy Friday, folks! Are the source rasters of a VRT added to the block cache such that different VRTs using the same source can avoid reads from disk or the network? Or is it intended that the VSI cache covers this need instead? Thanks, -- Sean Gillies -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Block cache and VRT sources
Even, Does the shared attribute of a VRT sourceFilename element not affect caching at all? Is the cache avoided so that potentially stale data isn't propagated, or for other reasons? On Fri, Apr 19, 2024 at 9:09 AM Even Rouault wrote: > Sean, > > Within a given GDALDataset opened on a VRT, if the VRT references several > times the same source, only one GDALDataset will be opened for it, so you > may benefit from the block cache mechanism (if it is triggered. > VRTSource::IRasterIO() calls IRasterIO() on the source band, which doesn't > necessarily always trigger block-based reading). But if you open another > VRT (or the same one), it will not share the same GDALDataset for sources > that may be common with the first one, so no re-use of existing block > cache. For network sources, the I/O cache at the /vsicurl/ level works > however on filenames, not VSIFILE* instances, so you will save network reads > > Even > Le 19/04/2024 à 16:48, Sean Gillies via gdal-dev a écrit : > > Happy Friday, folks! > > Are the source rasters of a VRT added to the block cache such that > different VRTs using the same source can avoid reads from disk or the > network? Or is it intended that the VSI cache covers this need instead? > > Thanks, > > -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Block cache and VRT sources
Sean, Within a given GDALDataset opened on a VRT, if the VRT references several times the same source, only one GDALDataset will be opened for it, so you may benefit from the block cache mechanism (if it is triggered. VRTSource::IRasterIO() calls IRasterIO() on the source band, which doesn't necessarily always trigger block-based reading). But if you open another VRT (or the same one), it will not share the same GDALDataset for sources that may be common with the first one, so no re-use of existing block cache. For network sources, the I/O cache at the /vsicurl/ level works however on filenames, not VSIFILE* instances, so you will save network reads Even Le 19/04/2024 à 16:48, Sean Gillies via gdal-dev a écrit : Happy Friday, folks! Are the source rasters of a VRT added to the block cache such that different VRTs using the same source can avoid reads from disk or the network? Or is it intended that the VSI cache covers this need instead? Thanks, -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Block cache and VRT sources
Happy Friday, folks! Are the source rasters of a VRT added to the block cache such that different VRTs using the same source can avoid reads from disk or the network? Or is it intended that the VSI cache covers this need instead? Thanks, -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
