Re: [gdal-dev] Serve COG images - GEE & Google Cloud Storage
Arun, For what it's worth, I routinely do this by preparing signed urls to the objects in GCS and just using normal /vsicurl/ references in mapfiles, etc. If you happen to be using boto libraries in Python signed urls can be created with the generate_url() call on "key" objects. Of course if you data is publish, you can just use the direct GCS urls without signing them. Best regards, Frank On Fri, Sep 4, 2020 at 8:38 AM Michael Smith wrote: > Another option is a fuse level driver that maps cloud storage to a drive > path. > > > > Mike > > > > > > -- > > Michael Smith > > US Army Corps of Engineers > > Remote Sensing/GIS Center > > > > > > *From: *gdal-dev on behalf of Travis > Kirstine > *Date: *Friday, September 4, 2020 at 8:35 AM > *To: *gdal dev > *Subject: *Re: [gdal-dev] Serve COG images - GEE & Google Cloud Storage > > > > > > There is a good article here on how to do this using MapServer and S3, > this may work for Google as well > > > > > https://github.com/mapserver/mapserver/wiki/Render-images-straight-out-of-S3-with-the-vsicurl-driver > > > > You can configure MapServer as a WMS server and add the layers to Open > Layers or take the extra step and configure MapCache to generate a tiled > output and caching using your WMS as source > > > > > > > > On Fri, 4 Sep 2020 at 03:01, Even Rouault > wrote: > > Arun, > > > > Not sure this completely answer your question, but GDAL has gained a > > Google Cloud Storage virtual file system handler similar > > to the AWS S3 one since the post you mention. > > See > https://gdal.org/user/virtual_file_systems.html#vsigs-google-cloud-storage-files > > > > Even > > > > > Hi > > > In this article, I came across about servicing rasters on AWS S3: > > > > https://www.azavea.com/blog/2019/04/23/using-cloud-optimized-geotiffs-cogs/ > > > , https://lists.osgeo.org/pipermail/gdal-dev/2015-October/042975.html > > > > > > I am trying to see if something similar exists for Google? I'm exporting > > > COG from Google Earth Engine (GEE) to a Google Cloud Storage bucket in > the > > > hopes to access and serve it with other layers using GeoServer or create > > > tile map services. I've many dates and many products (true color, false > > > color, ndvi) for many sites. So I'm not sure what's the best way to serve > > > them. I tried researching online, asking in StackExchange, GeoServer and > > > GEE forums and reaching out to a few people, but I couldn't get specific > > > info... Most info is on AWS and S3... > > > > > > My requirement is to somehow get the images of "GEE to bucket" exported > > > images into Openlayers (web mapping) and JavaScript (frontend). GEE can > > > export to my Drive or a GCP bucket. So serving from GEE to GCP to S3 > means > > > some download, unzip, zip and upload - so more time consuming and manual > > > steps. In my experience, AWS was costly (as my EC2 runs all the time) > than > > > a VPS server... > > > > > > I'll appreciate your help. Thank you! > > > > > > -- > > Spatialys - Geospatial professional services > > http://www.spatialys.com > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > > ___ gdal-dev mailing list > gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | +1 650-701-7823 and watch the world go round - Rush| Geospatial Software Developer ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Serve COG images - GEE & Google Cloud Storage
Another option is a fuse level driver that maps cloud storage to a drive path. Mike -- Michael Smith US Army Corps of Engineers Remote Sensing/GIS Center From: gdal-dev on behalf of Travis Kirstine Date: Friday, September 4, 2020 at 8:35 AM To: gdal dev Subject: Re: [gdal-dev] Serve COG images - GEE & Google Cloud Storage There is a good article here on how to do this using MapServer and S3, this may work for Google as well https://github.com/mapserver/mapserver/wiki/Render-images-straight-out-of-S3-with-the-vsicurl-driver You can configure MapServer as a WMS server and add the layers to Open Layers or take the extra step and configure MapCache to generate a tiled output and caching using your WMS as source On Fri, 4 Sep 2020 at 03:01, Even Rouault wrote: Arun, Not sure this completely answer your question, but GDAL has gained a Google Cloud Storage virtual file system handler similar to the AWS S3 one since the post you mention. See https://gdal.org/user/virtual_file_systems.html#vsigs-google-cloud-storage-files Even > Hi > In this article, I came across about servicing rasters on AWS S3: > https://www.azavea.com/blog/2019/04/23/using-cloud-optimized-geotiffs-cogs/ > , https://lists.osgeo.org/pipermail/gdal-dev/2015-October/042975.html > > I am trying to see if something similar exists for Google? I'm exporting > COG from Google Earth Engine (GEE) to a Google Cloud Storage bucket in the > hopes to access and serve it with other layers using GeoServer or create > tile map services. I've many dates and many products (true color, false > color, ndvi) for many sites. So I'm not sure what's the best way to serve > them. I tried researching online, asking in StackExchange, GeoServer and > GEE forums and reaching out to a few people, but I couldn't get specific > info... Most info is on AWS and S3... > > My requirement is to somehow get the images of "GEE to bucket" exported > images into Openlayers (web mapping) and JavaScript (frontend). GEE can > export to my Drive or a GCP bucket. So serving from GEE to GCP to S3 means > some download, unzip, zip and upload - so more time consuming and manual > steps. In my experience, AWS was costly (as my EC2 runs all the time) than > a VPS server... > > I'll appreciate your help. Thank you! -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Serve COG images - GEE & Google Cloud Storage
There is a good article here on how to do this using MapServer and S3, this may work for Google as well https://github.com/mapserver/mapserver/wiki/Render-images-straight-out-of-S3-with-the-vsicurl-driver You can configure MapServer as a WMS server and add the layers to Open Layers or take the extra step and configure MapCache to generate a tiled output and caching using your WMS as source On Fri, 4 Sep 2020 at 03:01, Even Rouault wrote: > Arun, > > > > Not sure this completely answer your question, but GDAL has gained a > > Google Cloud Storage virtual file system handler similar > > to the AWS S3 one since the post you mention. > > See > https://gdal.org/user/virtual_file_systems.html#vsigs-google-cloud-storage-files > > > > Even > > > > > Hi > > > In this article, I came across about servicing rasters on AWS S3: > > > > https://www.azavea.com/blog/2019/04/23/using-cloud-optimized-geotiffs-cogs/ > > > , https://lists.osgeo.org/pipermail/gdal-dev/2015-October/042975.html > > > > > > I am trying to see if something similar exists for Google? I'm exporting > > > COG from Google Earth Engine (GEE) to a Google Cloud Storage bucket in > the > > > hopes to access and serve it with other layers using GeoServer or create > > > tile map services. I've many dates and many products (true color, false > > > color, ndvi) for many sites. So I'm not sure what's the best way to serve > > > them. I tried researching online, asking in StackExchange, GeoServer and > > > GEE forums and reaching out to a few people, but I couldn't get specific > > > info... Most info is on AWS and S3... > > > > > > My requirement is to somehow get the images of "GEE to bucket" exported > > > images into Openlayers (web mapping) and JavaScript (frontend). GEE can > > > export to my Drive or a GCP bucket. So serving from GEE to GCP to S3 > means > > > some download, unzip, zip and upload - so more time consuming and manual > > > steps. In my experience, AWS was costly (as my EC2 runs all the time) > than > > > a VPS server... > > > > > > I'll appreciate your help. Thank you! > > > > > > -- > > Spatialys - Geospatial professional services > > http://www.spatialys.com > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Serve COG images - GEE & Google Cloud Storage
Arun, Not sure this completely answer your question, but GDAL has gained a Google Cloud Storage virtual file system handler similar to the AWS S3 one since the post you mention. See https://gdal.org/user/virtual_file_systems.html#vsigs-google-cloud-storage-files Even > Hi > In this article, I came across about servicing rasters on AWS S3: > https://www.azavea.com/blog/2019/04/23/using-cloud-optimized-geotiffs-cogs/ > , https://lists.osgeo.org/pipermail/gdal-dev/2015-October/042975.html > > I am trying to see if something similar exists for Google? I'm exporting > COG from Google Earth Engine (GEE) to a Google Cloud Storage bucket in the > hopes to access and serve it with other layers using GeoServer or create > tile map services. I've many dates and many products (true color, false > color, ndvi) for many sites. So I'm not sure what's the best way to serve > them. I tried researching online, asking in StackExchange, GeoServer and > GEE forums and reaching out to a few people, but I couldn't get specific > info... Most info is on AWS and S3... > > My requirement is to somehow get the images of "GEE to bucket" exported > images into Openlayers (web mapping) and JavaScript (frontend). GEE can > export to my Drive or a GCP bucket. So serving from GEE to GCP to S3 means > some download, unzip, zip and upload - so more time consuming and manual > steps. In my experience, AWS was costly (as my EC2 runs all the time) than > a VPS server... > > I'll appreciate your help. Thank you! -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Serve COG images - GEE & Google Cloud Storage
Hi In this article, I came across about servicing rasters on AWS S3: https://www.azavea.com/blog/2019/04/23/using-cloud-optimized-geotiffs-cogs/ , https://lists.osgeo.org/pipermail/gdal-dev/2015-October/042975.html I am trying to see if something similar exists for Google? I'm exporting COG from Google Earth Engine (GEE) to a Google Cloud Storage bucket in the hopes to access and serve it with other layers using GeoServer or create tile map services. I've many dates and many products (true color, false color, ndvi) for many sites. So I'm not sure what's the best way to serve them. I tried researching online, asking in StackExchange, GeoServer and GEE forums and reaching out to a few people, but I couldn't get specific info... Most info is on AWS and S3... My requirement is to somehow get the images of "GEE to bucket" exported images into Openlayers (web mapping) and JavaScript (frontend). GEE can export to my Drive or a GCP bucket. So serving from GEE to GCP to S3 means some download, unzip, zip and upload - so more time consuming and manual steps. In my experience, AWS was costly (as my EC2 runs all the time) than a VPS server... I'll appreciate your help. Thank you! -- Sincerely yours, *Arun Govind, PhD* *Email*: arun.gov...@usask.ca *Ph*: +16394714525 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev