Thanks so much for the response and I'm excited to hear that there is interest. Does any of the existing GDAL code specifically support COGs? My imagery skills aren't very strong, but I would hope that we would be able to use the request geometry to only request the specific range of bytes required. I'm assuming this is the part that would be handled by gdal itself, however I don't know what is necessary to make the request. Looking at these sample requests: https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF I would assume we'd have to convert the geo coords into the pixel space of the image, then pass those coords in the request (to translate? warp?) and gdal would know which ranges to request.
Not that this is any sort of revelation, but building the right Format implementation will also be important. I've found that I can pass an http url as a String directly to GeoTiffReader and it will read it fine, however when used as part of an ImageMosaic, it gets converted to a URL and the GeoTiffFormat class will throw an exception if the URL is anything other than a URL to a file. In this instance, I've had to provide a custom GeoTiffFormat implementation to not error out if the URL doesn't point to a file, and continue to build the GeoTiffReader and pass along the string value of the URL. The current s3-geotiff reader reads byte chunks from s3 and does not support COG. It also uses a non-standard format where the protocol is s3, and it strips out the last 2 parts of the path to determine the bucket and key. Then it relies on configuration to determine the region, etc. The Azure GeoTIFF reader I built follows the exact same code structure but implements the Azure API, but allows azure-specific "wasb://" and "wasbs://" protocols in the URLs. I was well on my way to creating a custom "HttpGeoTiffReader" when I started encountering the issues that started this thread. All that is to say, none of that seems optimal and building support for VFS would mean all of these can be replaced, as it supports all major cloud providers and would have baked-in COG support. Pretty exciting :) Additionally, it would be interesting to support some lightweight in-memory caching (or potentially external caching?) in the same way the s3-geotiff reader uses ehcache. And finally, support for async requests to assist when there may be multiple concurrent images/granules being requested. Thanks! Josh On Wed, Jun 26, 2019 at 10:13 AM Daniele Romagnoli < daniele.romagn...@geo-solutions.it> wrote: > Hi Josh, > I might be back with some more helpful feedbacks in the next few days but > I wanted to provide at least my reply right now, since I started > ImageIO-EXT/GeoTools/GeoServer - GDAL support/integration many years ago. > I think that supporting VFS would be a great and interesting contribution > to the project. > I need to check back the code more in detail since it's a couple of years > I didn't touch it, so I don't have precise feedbacks right now :) > Anyway, I think that a couple of key points are: > - updating the ImageIO-EXT low level reader machinery to handle that new > type of input. ImageReaders have a "setInput" method accepting an object. > The ImageReaderSPI will tell which types of input objects are supported. > SPI also have a canDecodeInput method which tell if the provided input can > be decoded. I think that the currently supported inputs are Files, URLs, > Strings (representing URL or Files), and FileImageInputStream*. Note that, > in the past, we also had to support an ECWP protocol as format requirement, > so we had to setup an ECWP input stream. However, I think that it has never > being used within GeoTools/GeoServer in practice. > - updating the GeoTools GDAL base classes. As you said, the > RasterLayerResponse is trying to setup a FileImageInputStreamExt because > the 99% of the GDAL ImageIO-Ext readers use that. However, if my memory > serves me right, the base GridCoverage2DReader abstract classes accept a > generic Object input (as well as the ImageReader as said before) so we may > "relax" the implementation to also support the VFS. > > As I said, sorry for these "generic" feedbacks. Hope they can be used as a > starting point for further investigations. > > Best Regards, > Daniele > > > > On Tue, Jun 25, 2019 at 5:36 PM Josh Fix via GeoTools-Devel < > geotools-devel@lists.sourceforge.net> wrote: > >> Hi all. I have code that builds an ImageMosaicReader that utilizes a >> custom format and input stream SPI. I define the classes I want to use in >> the Properties class used by the GranuleCatalog and in the >> CatalogConfigurationBean used by the >> ImageMosaicDescriptor/ImageMosaicReader. Generally, it looks something >> like this: >> >> props.put(Utils.Prop.SUGGESTED_IS_SPI, >> MyImageInputStreamSpi.class.getCanonicalName()); >> props.put(Utils.Prop.SUGGESTED_FORMAT, >> MyGeoTiffFormat.class.getCanonicalName()); >> props.put(Utils.Prop.SUGGESTED_SPI, >> "it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReaderSpi"); >> >> This has worked great in the past using custom input stream classes to >> read from S3 and Azure. My current goal is to be able to create mosaics >> from any given http endpoint, but specifically focus on cloud optimized >> geotiffs ( >> https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF#HowtoreaditwithGDAL) >> using the "vsicurl" VFS. I've written GDAL code in the past to open >> geotiffs on S3 using the /vsis3 VFS and it was incredibly efficient, so I >> sought out to do the same in GeoTools for my mosaic. >> >> I was delighted to discover many modules supporting GDAL readers. I >> started to experiment with >> https://github.com/geosolutions-it/imageio-ext/blob/master/plugin/gdal/gdalgeotiff/src/main/java/it/geosolutions/imageio/plugins/geotiff/GeoTiffImageReader.java >> and >> created my own Format implementation using the packages in >> https://github.com/geotools/geotools/tree/master/modules/plugin/imageio-ext-gdal/src/main/java/org/geotools/coverageio/gdal >> as >> templates. I created a simple GeoTiffReader class that extends from >> BaseGdalReader, which is responsible for creating the GeoTiffImageReaderSpi >> and GeoTiffFormat instances. >> >> A VFS URL takes the form: "/vsicurl/ >> https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/153/075/LC08_L1TP_153075_20190515_20190515_01_RT/LC08_L1TP_153075_20190515_20190515_01_RT_B3.TIF";... >> simply taking the http/https url an prefixing it with "/vsicurl/". This >> string can be passed directly to `gdal.Open()` and it will return a Dataset. >> >> The issue is that the input object that ends up being passed to >> BaseGridCoverage2DReader ends up expecting a file, either in the form of a >> URL, FileInputStreamExt, or String (see >> BaseGridCoverage2DReader.checkSource). I thought I might be able to trick >> it by passing in the vsicurl path prefixed with the file protocol >> (file:///vsicurl/https://s3-us-west...). The issue is that when Java >> returns the file path, it removes the second forward slash from https://, >> leaving you with the path "/vsicurl/https:/s3-us-west...", which gdal does >> not like. >> >> I duplicated a sufficient number of GeoTools classes to be able to add in >> simple string replacements for "https:/" -> "https://"; where needed. >> This allowed me to create the reader and successfully create the metadata >> object (in GDALImageReader.setInput). The BseGridCoverage2DReader >> constructor completes fully and all of the coverage properties, layout, >> resolution info, etc is built. At this point I was very hopeful that the >> only change necessary was modifying these classes to not only support >> files, but also strings that begin with "/vsi". >> >> Unfortunately this idea fell apart when I called "reader.read()". Long >> story short, the RasterLayerResponse object tries to create >> a FileImageInputStreamExtImpl, so it's not just reading the dataset using >> `gdal.Open`. At this point I'm not 100% certain what the best path forward >> would be. I know conversations in the past have centered around adding and >> supporting custom protocols to java.net.URL, but this is unique in that >> these "paths" are not valid URLs or file paths due to the double forward >> slash in the middle of the path. >> >> I would like to know if supporting VFS is something the community is >> interested in. I intend to continue to work on a solution for myself, but >> would love to work with you all and contribute back if there is interest. >> >> I apologize for the long read. I hope everything makes sense and please >> let me know if there are any questions. >> >> Josh >> _______________________________________________ >> GeoTools-Devel mailing list >> GeoTools-Devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/geotools-devel >> > > > -- > Regards, > Daniele Romagnoli > == > GeoServer Professional Services from the experts! Visit > http://goo.gl/it488V for more information. > == > > Ing. Daniele Romagnoli > Senior Software Engineer > > GeoSolutions S.A.S. > Via di Montramito 3/A > 55054 Massarosa (LU) > Italy > phone: +39 0584 962313 > fax: +39 0584 1660272 > > http://www.geo-solutions.it > http://twitter.com/geosolutions_it > > ------------------------------------------------------- > > Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE > 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si > precisa che ogni circostanza inerente alla presente email (il suo > contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è > riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il > messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra > operazione è illecita. Le sarei comunque grato se potesse darmene notizia. > > This email is intended only for the person or entity to which it is > addressed and may contain information that is privileged, confidential or > otherwise protected from disclosure. We remind that - as provided by > European Regulation 2016/679 “GDPR” - copying, dissemination or use of this > e-mail or the information herein by anyone other than the intended > recipient is prohibited. If you have received this email by mistake, please > notify us immediately by telephone or e-mail. >
_______________________________________________ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
