Hi Josh, I might be back with some more helpful feedbacks in the next few days but I wanted to provide at least my reply right now, since I started ImageIO-EXT/GeoTools/GeoServer - GDAL support/integration many years ago. I think that supporting VFS would be a great and interesting contribution to the project. I need to check back the code more in detail since it's a couple of years I didn't touch it, so I don't have precise feedbacks right now :) Anyway, I think that a couple of key points are: - updating the ImageIO-EXT low level reader machinery to handle that new type of input. ImageReaders have a "setInput" method accepting an object. The ImageReaderSPI will tell which types of input objects are supported. SPI also have a canDecodeInput method which tell if the provided input can be decoded. I think that the currently supported inputs are Files, URLs, Strings (representing URL or Files), and FileImageInputStream*. Note that, in the past, we also had to support an ECWP protocol as format requirement, so we had to setup an ECWP input stream. However, I think that it has never being used within GeoTools/GeoServer in practice. - updating the GeoTools GDAL base classes. As you said, the RasterLayerResponse is trying to setup a FileImageInputStreamExt because the 99% of the GDAL ImageIO-Ext readers use that. However, if my memory serves me right, the base GridCoverage2DReader abstract classes accept a generic Object input (as well as the ImageReader as said before) so we may "relax" the implementation to also support the VFS.
As I said, sorry for these "generic" feedbacks. Hope they can be used as a starting point for further investigations. Best Regards, Daniele On Tue, Jun 25, 2019 at 5:36 PM Josh Fix via GeoTools-Devel < geotools-devel@lists.sourceforge.net> wrote: > Hi all. I have code that builds an ImageMosaicReader that utilizes a > custom format and input stream SPI. I define the classes I want to use in > the Properties class used by the GranuleCatalog and in the > CatalogConfigurationBean used by the > ImageMosaicDescriptor/ImageMosaicReader. Generally, it looks something > like this: > > props.put(Utils.Prop.SUGGESTED_IS_SPI, > MyImageInputStreamSpi.class.getCanonicalName()); > props.put(Utils.Prop.SUGGESTED_FORMAT, > MyGeoTiffFormat.class.getCanonicalName()); > props.put(Utils.Prop.SUGGESTED_SPI, > "it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReaderSpi"); > > This has worked great in the past using custom input stream classes to > read from S3 and Azure. My current goal is to be able to create mosaics > from any given http endpoint, but specifically focus on cloud optimized > geotiffs ( > https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF#HowtoreaditwithGDAL) > using the "vsicurl" VFS. I've written GDAL code in the past to open > geotiffs on S3 using the /vsis3 VFS and it was incredibly efficient, so I > sought out to do the same in GeoTools for my mosaic. > > I was delighted to discover many modules supporting GDAL readers. I > started to experiment with > https://github.com/geosolutions-it/imageio-ext/blob/master/plugin/gdal/gdalgeotiff/src/main/java/it/geosolutions/imageio/plugins/geotiff/GeoTiffImageReader.java > and > created my own Format implementation using the packages in > https://github.com/geotools/geotools/tree/master/modules/plugin/imageio-ext-gdal/src/main/java/org/geotools/coverageio/gdal > as > templates. I created a simple GeoTiffReader class that extends from > BaseGdalReader, which is responsible for creating the GeoTiffImageReaderSpi > and GeoTiffFormat instances. > > A VFS URL takes the form: "/vsicurl/ > https://s3-us-west-2.amazonaws.com/landsat-pds/c1/L8/153/075/LC08_L1TP_153075_20190515_20190515_01_RT/LC08_L1TP_153075_20190515_20190515_01_RT_B3.TIF";... > simply taking the http/https url an prefixing it with "/vsicurl/". This > string can be passed directly to `gdal.Open()` and it will return a Dataset. > > The issue is that the input object that ends up being passed to > BaseGridCoverage2DReader ends up expecting a file, either in the form of a > URL, FileInputStreamExt, or String (see > BaseGridCoverage2DReader.checkSource). I thought I might be able to trick > it by passing in the vsicurl path prefixed with the file protocol > (file:///vsicurl/https://s3-us-west...). The issue is that when Java > returns the file path, it removes the second forward slash from https://, > leaving you with the path "/vsicurl/https:/s3-us-west...", which gdal does > not like. > > I duplicated a sufficient number of GeoTools classes to be able to add in > simple string replacements for "https:/" -> "https://"; where needed. > This allowed me to create the reader and successfully create the metadata > object (in GDALImageReader.setInput). The BseGridCoverage2DReader > constructor completes fully and all of the coverage properties, layout, > resolution info, etc is built. At this point I was very hopeful that the > only change necessary was modifying these classes to not only support > files, but also strings that begin with "/vsi". > > Unfortunately this idea fell apart when I called "reader.read()". Long > story short, the RasterLayerResponse object tries to create > a FileImageInputStreamExtImpl, so it's not just reading the dataset using > `gdal.Open`. At this point I'm not 100% certain what the best path forward > would be. I know conversations in the past have centered around adding and > supporting custom protocols to java.net.URL, but this is unique in that > these "paths" are not valid URLs or file paths due to the double forward > slash in the middle of the path. > > I would like to know if supporting VFS is something the community is > interested in. I intend to continue to work on a solution for myself, but > would love to work with you all and contribute back if there is interest. > > I apologize for the long read. I hope everything makes sense and please > let me know if there are any questions. > > Josh > _______________________________________________ > GeoTools-Devel mailing list > GeoTools-Devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geotools-devel > -- Regards, Daniele Romagnoli == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Daniele Romagnoli Senior Software Engineer GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
_______________________________________________ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
