Re: [gdal-dev] Understanding VSI caching
On lundi 9 avril 2018 21:03:12 CEST Andrea Aime wrote: > On Mon, Apr 9, 2018 at 8:58 PM, Even Rouault > > wrote: > > Which disk caching are you refering too ? There's some disk caching for > > the > > WMS driver, but that's all > > "In addition, a global least-recently-used cache of 16 MB shared among all > downloaded content is enabled by default, and content in it may be reused > after a file handle has been closed and reopen. Starting with GDAL 2.3, the > size of this global LRU cache can be modified by setting the configuration > option CPL_VSIL_CURL_CACHE_SIZE (in bytes)." Ah, it probably means a precision "during the life-time of the process or until VSICurlClearCache() is called" (just added). It's just a RAM cache. > > and I see in the code: > > https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_curl.cpp#L203 > > (btw, there is no path, does it mean the file is created in the current > directory?) Yes, that was the more or less dead code I was mentionning too. Actually it is compiled in, and can be enabled with CPL_VSIL_CURL_USE_CACHE=YES, but it is very likely broken, unfit for any reasonable purpose etc. No warranty. Consult your lawyer, be sure you have a fallout shelter ready, etc ;-) Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Understanding VSI caching
On Mon, Apr 9, 2018 at 8:58 PM, Even Rouault wrote: > Which disk caching are you refering too ? There's some disk caching for the > WMS driver, but that's all > > "In addition, a global least-recently-used cache of 16 MB shared among all downloaded content is enabled by default, and content in it may be reused after a file handle has been closed and reopen. Starting with GDAL 2.3, the size of this global LRU cache can be modified by setting the configuration option CPL_VSIL_CURL_CACHE_SIZE (in bytes)." and I see in the code: https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_curl.cpp#L203 (btw, there is no path, does it mean the file is created in the current directory?) > (well there's some provisional code for disk caching in /vsicurl/ but it is > not compiled by default, and probably broken. There was some discussion > recently on the ML about this if I remember well) > I'll check the archives. Cheers Andrea == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Understanding VSI caching
> I read there was some caching, but was not that specific. So, memory > caching is there, it's per file, > but not enabled by default, Yes > while disk caching is there, global, and > enabled by default? Which disk caching are you refering too ? There's some disk caching for the WMS driver, but that's all (well there's some provisional code for disk caching in /vsicurl/ but it is not compiled by default, and probably broken. There was some discussion recently on the ML about this if I remember well) > > The "detect sequential reading" part is interesting, if I'm not mistaken > the logic is here, > right? > > https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_curl.cpp#L1845 Yes -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Understanding VSI caching
On Mon, Apr 9, 2018 at 6:43 PM, Even Rouault wrote: > Hi Andrea, > > Are you speaking about the cache machanism enabled with the VSI_CACHE=YES > env variable ? > I read there was some caching, but was not that specific. So, memory caching is there, it's per file, but not enabled by default, while disk caching is there, global, and enabled by default? > If so that's described at > http://gdal.org/gdal_virtual_file_systems.html#gdal_ > virtual_file_systems_vsicache > I see it lacks a bit high level details, like specifying this is RAM only, > and not disk cached. > I'm going to add some words about that > > Otherwise, the full "documentation" :-) is at: > https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_cache.cpp Hehehe :-) > The /vsicurl/ and related network file systems have also their own RAM > cache mechanism > described at > http://gdal.org/gdal_virtual_file_systems.html#gdal_ > virtual_file_systems_vsicurl The "detect sequential reading" part is interesting, if I'm not mistaken the logic is here, right? https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_curl.cpp#L1845 Thanks a lot for the useful information, as usual :-) Cheers Andrea == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Understanding VSI caching
On lundi 9 avril 2018 18:33:32 CEST Andrea Aime wrote: > Hi, > is there any document explaning how VSI caching works. E.g., memory vs disk > caching, whether the disk cache > is long lived or exists only for the duration of the process (and if so, > what about long lived processes > like mapserver fastcgi?) and so on? Hi Andrea, Are you speaking about the cache machanism enabled with the VSI_CACHE=YES env variable ? If so that's described at http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsicache I see it lacks a bit high level details, like specifying this is RAM only, and not disk cached. I'm going to add some words about that Otherwise, the full "documentation" :-) is at: https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_cache.cpp The /vsicurl/ and related network file systems have also their own RAM cache mechanism described at http://gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsicurl Even > > Cheers > Andrea > > == > > GeoServer Professional Services from the experts! Visit http://goo.gl/it488V > for more information. > == > > Ing. Andrea Aime > @geowolf > Technical Lead > > GeoSolutions S.A.S. > Via di Montramito 3/A > 55054 Massarosa (LU) > phone: +39 0584 962313 > fax: +39 0584 1660272 > mob: +39 339 8844549 > > http://www.geo-solutions.it > http://twitter.com/geosolutions_it > > AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 > > Le informazioni contenute in questo messaggio di posta elettronica e/o > nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il > loro utilizzo è consentito esclusivamente al destinatario del messaggio, > per le finalità indicate nel messaggio stesso. Qualora riceviate questo > messaggio senza esserne il destinatario, Vi preghiamo cortesemente di > darcene notizia via e-mail e di procedere alla distruzione del messaggio > stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, > divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od > utilizzarlo per finalità diverse, costituisce comportamento contrario ai > principi dettati dal D.Lgs. 196/2003. > > The information in this message and/or attachments, is intended solely for > the attention and use of the named addressee(s) and may be confidential or > proprietary in nature or covered by the provisions of privacy act > (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection > Code).Any use not in accord with its purpose, any disclosure, reproduction, > copying, distribution, or either dissemination, either whole or partial, is > strictly forbidden except previous formal approval of the named > addressee(s). If you are not the intended recipient, please contact > immediately the sender by telephone, fax or e-mail and delete the > information in this message that has been received in error. The sender > does not give any warranty or accept liability as the content, accuracy or > completeness of sent messages and accepts no responsibility for changes > made after they were sent or for other risks which arise as a result of > e-mail transmission, viruses, etc. -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Understanding VSI caching
Hi, is there any document explaning how VSI caching works. E.g., memory vs disk caching, whether the disk cache is long lived or exists only for the duration of the process (and if so, what about long lived processes like mapserver fastcgi?) and so on? Cheers Andrea == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
