Re: [mapserver-users] Storing GeoTIFFs in Amazon S3 for use with Mapserver?
Robert, I was actually using the vsicurl support from a prep-script similar to gdaltindex, not from in mapserver directly, so I don't have a handy example. I'd also be concerned that accessing the data via /vsicurl/ would have performance problems that would make it unwise for direct access from MapServer. One benefit of the S3 file system handlers is that they cache the files locally so multiple tile renders in mapserver from the file would be fast after the first fetch. If you use /vsicurl/ then I don't think that local caching would occur between mapserver render requests. Best regards, Frank On Wed, Nov 5, 2014 at 12:07 PM, Robert Sanson < robert.san...@asurequality.com> wrote: > Thanks Frank. > > Do you have an example of a mapfile layer section using GDAL with vsicurl > to retrieve GeoTIFFs? > > In my case I use a tileindex to access a directory of GeoTIFFs but we want > to move them into S3. Here is a layer from my mapfile: > > LAYER > NAME sthl-075m-rural-2005-11 > GROUP 'highres' > TYPE RASTER > STATUS ON > TILEINDEX ../../data/nz/sthl-075m-rural-2005-11_ind > TILEITEM Location > MAXSCALE 10 > METADATA >'ows_title' 'sthl-075m-rural-2005-11' >'wms_opaque' '1' >'wms_group_title' 'highres' >'wms_resx' '0.75' >'wms_resy' '0.75' >'ows_extent' '1088800.0 4743600.0 1314400.0 5089200.0' > END > END > > Thanks, > > Robert > > >>> Frank Warmerdam 5/11/2014 1:45 p.m. >>> > Robert, > > I've done some variations on this. > > I'd encourage you to watch a video by Mark Korver on this topic from > FOSS4G: http://vimeo.com/106833852 > > I used his NAIP bucket in "requestor-pays" mode successfully. One thing > that I ended up doing when reading the metadata was to use the vsicurl > driver to access the tiffs, so only the required portions were fetched. > His example actually mounts the whole bucket as a file system in which case > things are much easier. > > I will note that when using vsicurl to access GeoTIFFs in S3 I found I had > to set the following configuration variables to get it working. > > # Various /vsicurl hackery to get it working with signed aws urls well > gdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS', 'ter') > gdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN', 'YES') > gdal.SetConfigOption('CPL_VSIL_CURL_USE_HEAD', 'NO') > > I do quite a bit of stuff here with MapServer from S3, but I usually fetch > from S3 and store in /tmp for serving so by the time MapServer sees the > file it is completely normal. > > Best regards, > Frank > > > On Tue, Nov 4, 2014 at 2:56 PM, Robert Sanson < > robert.san...@asurequality.com> wrote: > > > Does anyone have experience with using Amazon S3 to store GeoTIffs for > use > > as a back-end datastore accessible by Mapserver in order to create a WMS > > service? > > > > Thanks, > > > > Robert Sanson > > > > > > Robert Sanson Veterinary Epidemiologist > > Batchelar Centre | Tennent Drive | Palmerston North > > PO Box 585 | Palmerston North 4440 | New Zealand > > p. +64 6 351 7990 | m. +64 21 448 472 | f. +64 6 351 7919 > > e. robert.san...@asurequality.com > > > > Global experts in food safety and quality | www.asurequality.com > > > > > > > > > > This email and any attachments are confidential and intended solely for > > the addressee(s). If you are not the intended recipient, please notify us > > immediately and then delete this email from your system. > > > > This message has been scanned for Malware and Viruses by Websense Hosted > > Security. > > www.websense.com > > ___ > > mapserver-users mailing list > > mapserver-users@lists.osgeo.org > > http://lists.osgeo.org/mailman/listinfo/mapserver-users > > > > > > -- > > ---+-- > I set the clouds in motion - turn up | Frank Warmerdam, > warmer...@pobox.com > light and sound - activate the windows | http://pobox.com/~warmerdam > and watch the world go round - Rush| Geospatial Software Developer > > -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Software Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Storing GeoTIFFs in Amazon S3 for use with Mapserver?
Thanks Frank. Do you have an example of a mapfile layer section using GDAL with vsicurl to retrieve GeoTIFFs? In my case I use a tileindex to access a directory of GeoTIFFs but we want to move them into S3. Here is a layer from my mapfile: LAYER NAME sthl-075m-rural-2005-11 GROUP 'highres' TYPE RASTER STATUS ON TILEINDEX ../../data/nz/sthl-075m-rural-2005-11_ind TILEITEM Location MAXSCALE 10 METADATA 'ows_title' 'sthl-075m-rural-2005-11' 'wms_opaque' '1' 'wms_group_title' 'highres' 'wms_resx' '0.75' 'wms_resy' '0.75' 'ows_extent' '1088800.0 4743600.0 1314400.0 5089200.0' END END Thanks, Robert >>> Frank Warmerdam 5/11/2014 1:45 p.m. >>> Robert, I've done some variations on this. I'd encourage you to watch a video by Mark Korver on this topic from FOSS4G: http://vimeo.com/106833852 I used his NAIP bucket in "requestor-pays" mode successfully. One thing that I ended up doing when reading the metadata was to use the vsicurl driver to access the tiffs, so only the required portions were fetched. His example actually mounts the whole bucket as a file system in which case things are much easier. I will note that when using vsicurl to access GeoTIFFs in S3 I found I had to set the following configuration variables to get it working. # Various /vsicurl hackery to get it working with signed aws urls well gdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS', 'ter') gdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN', 'YES') gdal.SetConfigOption('CPL_VSIL_CURL_USE_HEAD', 'NO') I do quite a bit of stuff here with MapServer from S3, but I usually fetch from S3 and store in /tmp for serving so by the time MapServer sees the file it is completely normal. Best regards, Frank On Tue, Nov 4, 2014 at 2:56 PM, Robert Sanson < robert.san...@asurequality.com> wrote: > Does anyone have experience with using Amazon S3 to store GeoTIffs for use > as a back-end datastore accessible by Mapserver in order to create a WMS > service? > > Thanks, > > Robert Sanson > > > Robert Sanson Veterinary Epidemiologist > Batchelar Centre | Tennent Drive | Palmerston North > PO Box 585 | Palmerston North 4440 | New Zealand > p. +64 6 351 7990 | m. +64 21 448 472 | f. +64 6 351 7919 > e. robert.san...@asurequality.com > > Global experts in food safety and quality | www.asurequality.com > > > > > This email and any attachments are confidential and intended solely for > the addressee(s). If you are not the intended recipient, please notify us > immediately and then delete this email from your system. > > This message has been scanned for Malware and Viruses by Websense Hosted > Security. > www.websense.com > ___ > mapserver-users mailing list > mapserver-users@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/mapserver-users > -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Software Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Storing GeoTIFFs in Amazon S3 for use with Mapserver?
Robert, I've done some variations on this. I'd encourage you to watch a video by Mark Korver on this topic from FOSS4G: http://vimeo.com/106833852 I used his NAIP bucket in "requestor-pays" mode successfully. One thing that I ended up doing when reading the metadata was to use the vsicurl driver to access the tiffs, so only the required portions were fetched. His example actually mounts the whole bucket as a file system in which case things are much easier. I will note that when using vsicurl to access GeoTIFFs in S3 I found I had to set the following configuration variables to get it working. # Various /vsicurl hackery to get it working with signed aws urls well gdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS', 'ter') gdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN', 'YES') gdal.SetConfigOption('CPL_VSIL_CURL_USE_HEAD', 'NO') I do quite a bit of stuff here with MapServer from S3, but I usually fetch from S3 and store in /tmp for serving so by the time MapServer sees the file it is completely normal. Best regards, Frank On Tue, Nov 4, 2014 at 2:56 PM, Robert Sanson < robert.san...@asurequality.com> wrote: > Does anyone have experience with using Amazon S3 to store GeoTIffs for use > as a back-end datastore accessible by Mapserver in order to create a WMS > service? > > Thanks, > > Robert Sanson > > > Robert Sanson Veterinary Epidemiologist > Batchelar Centre | Tennent Drive | Palmerston North > PO Box 585 | Palmerston North 4440 | New Zealand > p. +64 6 351 7990 | m. +64 21 448 472 | f. +64 6 351 7919 > e. robert.san...@asurequality.com > > Global experts in food safety and quality | www.asurequality.com > > > > > This email and any attachments are confidential and intended solely for > the addressee(s). If you are not the intended recipient, please notify us > immediately and then delete this email from your system. > > This message has been scanned for Malware and Viruses by Websense Hosted > Security. > www.websense.com > ___ > mapserver-users mailing list > mapserver-users@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/mapserver-users > -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Software Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Storing GeoTIFFs in Amazon S3 for use with Mapserver?
Does anyone have experience with using Amazon S3 to store GeoTIffs for use as a back-end datastore accessible by Mapserver in order to create a WMS service? Thanks, Robert Sanson Robert Sanson Veterinary Epidemiologist Batchelar Centre | Tennent Drive | Palmerston North PO Box 585 | Palmerston North 4440 | New Zealand p. +64 6 351 7990 | m. +64 21 448 472 | f. +64 6 351 7919 e. robert.san...@asurequality.com Global experts in food safety and quality | www.asurequality.com This email and any attachments are confidential and intended solely for the addressee(s). If you are not the intended recipient, please notify us immediately and then delete this email from your system. This message has been scanned for Malware and Viruses by Websense Hosted Security. www.websense.com ___ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users