Re: GeoDjango OffdbRasterField
Your analysis is spot on, PostGIS references the path to off-db rasters using an absolute path, which needs to be resolved on the PostGIS server. I guess a developer could map a folder (via samba for example) on the PostGIS server to match that of the Django server so that both servers share the same path structure. There would probably be some problems with this approach as well, especially when mixing different operating systems, or synchronization issues depending on the type of network mapping used. If this addition is too problematic to be added in core, I could just isolate the code and release it as a plugin, with some documentation about the limits of the approach. On Thursday, December 29, 2016 at 2:35:59 PM UTC-5, Daniel Wiesmann wrote: > > This is an interesting idea and approach, thanks Piero for the suggestion > and the proposed solution. > > I agree with Piero and Adam, it would be a great addition to the raster > field. Especially for large volumes of data (not only for individual large > files, but also for many small files). Raster data can eat up a lot of > storage quickly, and delegating that to an off-db storage is useful. > > I have been looking at the out-of-db feature for PostGIS rasters, but I am > not sure if it can be used in the Django context. The problem I see is the > storage location. > > If I understand it right, then the raster files need to be on the same > system as the PostGIS instance, i.e. they need to be available on the > filesystem of the postgis server. Like that, PostGIS can access the files > from within its raster operations (such as intersections, getting pixel > values etc), and it will work as if the data was stored in the db directly. > > In that case however, the out-of-db field would only work if Django is > running on the same server as the PostGIS instance. So I think it would > fail for more distributed systems, where you have multiple application > instances, and a remote PostGIS, which might have replicas etc. > > In your experience Piero, is that correct? How are the out-of-db raster > files stored by PostGIS? > > I have never used the out-of-db raster option so I am not sure if I > understood the storage mechanism correclty. > > Coincidently, I recently had a similar problem, and I have been testing a > version of a raster field which is a subclass of the regular Django > FileField. In this approach, the storage can be any Django compatible > storage (including remote object storages such as S3). The downside is that > PostGIS will no longer recognize the data as rasters, so lookups and > PostGIS internal functions will not be available. > > > This is still quite experimental (it uses a patched Django with an > extension of the GDALRaster) and is slightly hacky, but does convert the > remote rasters into GDALRaster instances when opened. > > > https://github.com/geodesign/django-raster/compare/raster_file_field > > > An ideal solution would be a mixture of the two, but I am not sure if > PostGIS can handle remote storages. > > On Tuesday, December 20, 2016 at 4:33:21 PM UTC, Tim Graham wrote: >> >> Daniel Wiesmann did all the work for RasterField. I'm not sure if he >> follows this list but you can find his email address in the Django commit >> longs and mail him to ask for his input. >> >> https://github.com/yellowcap >> >> On Tuesday, December 20, 2016 at 11:27:25 AM UTC-5, Piero Toffanin wrote: >>> >>> It's stored on the file system. This is to improve performance when >>> storing large geospatial datasets. >>> >>> This would only work on PostGIS. >>> >>> On Thursday, December 15, 2016 at 3:11:37 PM UTC-5, Adam Johnson wrote: >>>> >>>> I can't say I'm that familiar with GeoDjango, but that does sound like >>>> a useful feature. Where does the data get stored if not in the DB? And >>>> does >>>> this feature exist on any of the other database backends that GeoDjango >>>> supports? >>>> >>>> On 14 December 2016 at 18:40, Piero Toffanin <pier...@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> Not sure this is the right place to post this, if not, could somebody >>>>> point me to the right place? >>>>> >>>>> I recently had the need to use GeoDjango to define a model that uses a >>>>> RasterField to store a GeoTIFF raster. >>>>> https://docs.djangoproject.com/en/1.10/ref/contrib/gis/model-api/#rasterfield >>>>> >>>>> Upon testing, one of the users of the application found that loading a >
Re: GeoDjango OffdbRasterField
It's stored on the file system. This is to improve performance when storing large geospatial datasets. This would only work on PostGIS. On Thursday, December 15, 2016 at 3:11:37 PM UTC-5, Adam Johnson wrote: > > I can't say I'm that familiar with GeoDjango, but that does sound like a > useful feature. Where does the data get stored if not in the DB? And does > this feature exist on any of the other database backends that GeoDjango > supports? > > On 14 December 2016 at 18:40, Piero Toffanin <pier...@gmail.com > > wrote: > >> Hello, >> >> Not sure this is the right place to post this, if not, could somebody >> point me to the right place? >> >> I recently had the need to use GeoDjango to define a model that uses a >> RasterField to store a GeoTIFF raster. >> https://docs.djangoproject.com/en/1.10/ref/contrib/gis/model-api/#rasterfield >> >> Upon testing, one of the users of the application found that loading a >> large GeoTIFF (100Mb+) caused PostgreSQL to fail. More details about the >> report are available here: >> https://github.com/OpenDroneMap/WebODM/issues/55 >> >> I went around the problem by using a relatively new feature of PostGIS >> that allows raster files to be stored off db. I noticed that RasterField >> does not support such feature, so I wrote the code to enable support for it >> via a new OffdbRasterField ( >> https://github.com/OpenDroneMap/WebODM/blob/master/app/postgis.py). >> The from_pgraster and to_pgraster functions are modified versions of the >> same functions found here: >> https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py >> >> Just wanted to see if there was an interest in adding off db raster >> support into GeoDjango core. Perhaps by modifying RasterField to have an >> additional parameter "offdb=True|False" and implement the necessary logic? >> >> Thanks, >> >> -Piero >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-develop...@googlegroups.com . >> To post to this group, send email to django-d...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/django-developers. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com >> >> <https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com?utm_medium=email_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Adam > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5b0ba3e3-733f-4304-bacf-dfc027d64806%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
GeoDjango OffdbRasterField
Hello, Not sure this is the right place to post this, if not, could somebody point me to the right place? I recently had the need to use GeoDjango to define a model that uses a RasterField to store a GeoTIFF raster. https://docs.djangoproject.com/en/1.10/ref/contrib/gis/model-api/#rasterfield Upon testing, one of the users of the application found that loading a large GeoTIFF (100Mb+) caused PostgreSQL to fail. More details about the report are available here: https://github.com/OpenDroneMap/WebODM/issues/55 I went around the problem by using a relatively new feature of PostGIS that allows raster files to be stored off db. I noticed that RasterField does not support such feature, so I wrote the code to enable support for it via a new OffdbRasterField (https://github.com/OpenDroneMap/WebODM/blob/master/app/postgis.py). The from_pgraster and to_pgraster functions are modified versions of the same functions found here: https://github.com/django/django/blob/master/django/contrib/gis/db/backends/postgis/pgraster.py Just wanted to see if there was an interest in adding off db raster support into GeoDjango core. Perhaps by modifying RasterField to have an additional parameter "offdb=True|False" and implement the necessary logic? Thanks, -Piero -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/cbcf7634-9b24-418a-9c16-9e9b7f77fa51%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
