Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-19 Thread Even Rouault
Le jeudi 19 décembre 2013 04:41:18, Trent Piepho a écrit : > Do you see page file activity? If you look at /proc/pid/smaps, you > should be able to see the actual status of the mapping of your data > file. Probably it is consuming a large number of pages of RAM, but > also there should be zero pa

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-18 Thread Trent Piepho
Do you see page file activity? If you look at /proc/pid/smaps, you should be able to see the actual status of the mapping of your data file. Probably it is consuming a large number of pages of RAM, but also there should be zero pages written to swap. All clean private or clean shared, zero anony

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-18 Thread Even Rouault
Le mercredi 18 décembre 2013 21:09:48, Trent Piepho a écrit : > On Wed, Dec 18, 2013 at 11:46 AM, Even Rouault > > wrote: > > Le mercredi 18 décembre 2013 19:53:37, Frank Warmerdam a écrit : > >> I imagined an available virtual method on the band which could be > >> implemented - primarily by the

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-18 Thread Trent Piepho
On Wed, Dec 18, 2013 at 11:46 AM, Even Rouault wrote: > Le mercredi 18 décembre 2013 19:53:37, Frank Warmerdam a écrit : >> >> I imagined an available virtual method on the band which could be >> implemented - primarily by the RawBand class to try and mmap() the data and >> return the layout. But

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-18 Thread Frank Warmerdam
On Wed, Dec 18, 2013 at 11:46 AM, Even Rouault wrote: > > I'm wondering if there would be > > ways of making what you propose work with Python Numpy in such a way > that a > > numpy array could be requested which is of this virtual memory. That > would > > also be a nice extension. > > Hum, how

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-18 Thread Frank Warmerdam
On Wed, Dec 18, 2013 at 11:46 AM, Even Rouault wrote: > Le mercredi 18 décembre 2013 19:53:37, Frank Warmerdam a écrit : > > Even, > > > > Sorry, I was thinking of mmap() directly to the file, and having > something > > like: > > > > CPLVirtualMem CPL_DLL* GDALBandGetVirtualMemAuto( GDALRasterBan

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-18 Thread Even Rouault
Le mercredi 18 décembre 2013 19:53:37, Frank Warmerdam a écrit : > Even, > > Sorry, I was thinking of mmap() directly to the file, and having something > like: > > CPLVirtualMem CPL_DLL* GDALBandGetVirtualMemAuto( GDALRasterBandH hBand, > int *pnPixelSpace

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-18 Thread Frank Warmerdam
Even, Sorry, I was thinking of mmap() directly to the file, and having something like: CPLVirtualMem CPL_DLL* GDALBandGetVirtualMemAuto( GDALRasterBandH hBand, int *pnPixelSpace, GIntBig *pnLineSpace,

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-18 Thread Even Rouault
Le mercredi 18 décembre 2013 06:55:50, Frank Warmerdam a écrit : > Even, > > Very impressive work, I am supportive. > > IMHO it would be wonderful if there was also an mmap() based mechanism > where you could ask for the virtual memory chunk and you get it back (if it > works) along with stride v

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-17 Thread Frank Warmerdam
Even, Very impressive work, I am supportive. IMHO it would be wonderful if there was also an mmap() based mechanism where you could ask for the virtual memory chunk and you get it back (if it works) along with stride values to access in it. This could likely be made to work for most "raw" based

Re: [gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-17 Thread Even Rouault
Le mardi 17 décembre 2013 21:54:31, Even Rouault a écrit : > Hi, > > This is a call for discussion for "RFC 45: GDAL datasets and raster bands > as virtual memory mappings" Here's the link to the RFC : http://trac.osgeo.org/gdal/wiki/rfc45_virtualmem Even -- Geospatial professional services h

[gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

2013-12-17 Thread Even Rouault
Hi, This is a call for discussion for "RFC 45: GDAL datasets and raster bands as virtual memory mappings" Beginning of the RFC inline (the full RFC includes a few colorful schemas !) : """ == Summary == This document proposes additions to GDAL so that image data of GDAL datasets and raster b