Re: [Geoserver-devel] Geopackage tiles generation issues
Le mercredi 19 octobre 2016 12:12:03, Andrea Aime a écrit : > Hi Even, > thanks a lot for chiming in :-) > > I have a question about GDAL behavior, I made some more tests today and the > geopackage > always seems to have only one zoom level. Wondering if I just fussed around > yesterday or I used > some option that I don't remember about :-) > > Looking at the docs it seems that if someone wants some overviews gdaladdo > should be used to add them If you just do gdal_translate or gdalwarp, you will get only the maximum zoom level filled in (entries for lower zoom levels will be reserved in gpkg_tile_matrix though) If you want lower zoom levels to be filled, you indeed need to run gdaladdo. Exactly for other formats like GeoTIFF etc. Even -- Spatialys - Geospatial professional services http://www.spatialys.com -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Re: [Geoserver-devel] Geopackage tiles generation issues
Hi Even, thanks a lot for chiming in :-) I have a question about GDAL behavior, I made some more tests today and the geopackage always seems to have only one zoom level. Wondering if I just fussed around yesterday or I used some option that I don't remember about :-) Looking at the docs it seems that if someone wants some overviews gdaladdo should be used to add them Cheers Andrea -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. --- -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot___ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Re: [Geoserver-devel] Geopackage tiles generation issues
Le mardi 18 octobre 2016 12:13:59, Rahkonen Jukka (MML) a écrit : > Hi, > > >Given that the modules are unsupported and not working properly anyways, > >I'd >rather fix things to follow GDAL's approach. Any objection? > > No objection. If two open source GIS software under the OSGeo umbrella do > something in the same way we can start calling it as a de facto standard. In the matter that would be more like "profiles" of what is allowed within the OGC standard. > > Even has been committing a lot to GPKG lately but mostly to the vector > side. It might still be good to check if GDAL still follows the same > approach as you are planning to follow. No change in the GPKG raster approach on the GDAL side done recently. Andrea's analysis matches well how it is designed. Note that Brad Hards identified a possible non compliance of the GDAL writer in some circumstances regarding a picky requirement See (the part about Requirement 45, the issue with now vs CURRENT_TIMESTAMP has now been fixed) https://lists.osgeo.org/pipermail/gdal-dev/2016-August/045098.html https://lists.osgeo.org/pipermail/gdal-dev/2016-August/045100.html > > -Jukka Rahkonen- > > Andrea Aime wrote: > [Geoserver-devel] Geopackage tiles generation issues > > Hi, > I am looking into why the raster geopackages GeoServer generates via the > geopackage module are not being read correctly by gdal and found a few > issues, some straight bugs, other more like odd design decisions. > > The straight bug is what's causing GDAL not to read the files correctly, > the current code, shared with mbtiles generation, forces the request to be > cast on a GWC managed gridset (and actually works only if the requested > layers are cached, which is a bit of a nonsense to me given this is a WMS > request...), and uses that gridset coordinates for tiles. > Now the problem is that in the geopackage metadata for the layer and the > gridset we are declaring the requested bbox, not the ones used for tiles > generation, resulting in tile coordinates being off the expected grid. > Now, geopackage natively has the layer bounds that are separate from the > tile matrix ones, but the current geotools code forces them to be the > same.. that's a limitation I'm fixing. > > However, the main issue remains... if someone asks for a layer in a certain > SRS, why try to coax it into a tile matrix managed by GWC? > I believe the intention was to try and leverage GWC cache, but still, while > I understand taking the opportunity if possible, in my opinion the request > of the user should be respected, and if the request is not a match either > by SRS or BBOX to a tileset, GeoServer should not force a different > response. > From the same line of thought, a WMS request should not fail if it's made > against a layer that is not cached by GWC, like it does today, with a > strange error message too, e.g. asking for UTM 32N, which is not tile > cached: > > http://demo.geo-solutions.it/geoserver/wms/reflect?layers=nurc:mosaic&srs=E > PSG:32632&format=geopackage > > results in: > > java.lang.IllegalArgumentException: Thread 19276 Unknown layer nurc:mosaic. > Check the logfiles, it may not have loaded properly > > The second thing that I find confusing is, if I asked for a result image > that's 768 pixels wide, why am I getting back a geopackage that's over > 10 pixels wide? That's... insane, especially for a WMS request (large > jobs should be deferred to WPS asynch requests). Looking at the code, > there is logic that generates the min and max zoom levels for the tile > matrix, and it does something rather surprising... it picks the zoom level > closest to the requested one as the min zoom level, and then starts adding > zoom levels until the matrix has (stricly) more than 256 tiles in width... > in my case, that means 512 tiles, or a total width of 512*256=131072 > pixels! > > Trying GDAL on the command line, translating from a geotiff to a > geopackage, it makes what looks like a more sensible choice instead: it > uses the requested zoom level as the _maximum_ one, and then adds > overviews to it, in the cases I've tried, until a single tile is left > (basically, a good output with overviews). With our case we can also > generate raster geopackages out of vector layers, and creating the > overviews might end up hitting badly setup styles displaying too much > data, but still, that would be a problem also in random WMS requests, so > I'm not too concerned... > > Also, by default, GDAL generates a gridset that's an exact match for the > source data, meaning a custom one. That also makes sense to me, if someone > wants something other than the BBOX/SRS requested there are format options > to choose both the target gridset and min/max zoom levels. > > Given that the modules are unsupported and not working properly anyways, > I'd rather fix things to follow GDAL's approach. Any objection? > > Cheers > Andrea > > > -- > == > GeoServer Professional Services from the expe
Re: [Geoserver-devel] Geopackage tiles generation issues
Hi, >Given that the modules are unsupported and not working properly anyways, I'd >>rather fix things to follow GDAL's approach. >Any objection? No objection. If two open source GIS software under the OSGeo umbrella do something in the same way we can start calling it as a de facto standard. Even has been committing a lot to GPKG lately but mostly to the vector side. It might still be good to check if GDAL still follows the same approach as you are planning to follow. -Jukka Rahkonen- Andrea Aime wrote: [Geoserver-devel] Geopackage tiles generation issues Hi, I am looking into why the raster geopackages GeoServer generates via the geopackage module are not being read correctly by gdal and found a few issues, some straight bugs, other more like odd design decisions. The straight bug is what's causing GDAL not to read the files correctly, the current code, shared with mbtiles generation, forces the request to be cast on a GWC managed gridset (and actually works only if the requested layers are cached, which is a bit of a nonsense to me given this is a WMS request...), and uses that gridset coordinates for tiles. Now the problem is that in the geopackage metadata for the layer and the gridset we are declaring the requested bbox, not the ones used for tiles generation, resulting in tile coordinates being off the expected grid. Now, geopackage natively has the layer bounds that are separate from the tile matrix ones, but the current geotools code forces them to be the same.. that's a limitation I'm fixing. However, the main issue remains... if someone asks for a layer in a certain SRS, why try to coax it into a tile matrix managed by GWC? I believe the intention was to try and leverage GWC cache, but still, while I understand taking the opportunity if possible, in my opinion the request of the user should be respected, and if the request is not a match either by SRS or BBOX to a tileset, GeoServer should not force a different response. From the same line of thought, a WMS request should not fail if it's made against a layer that is not cached by GWC, like it does today, with a strange error message too, e.g. asking for UTM 32N, which is not tile cached: http://demo.geo-solutions.it/geoserver/wms/reflect?layers=nurc:mosaic&srs=EPSG:32632&format=geopackage results in: java.lang.IllegalArgumentException: Thread 19276 Unknown layer nurc:mosaic. Check the logfiles, it may not have loaded properly The second thing that I find confusing is, if I asked for a result image that's 768 pixels wide, why am I getting back a geopackage that's over 10 pixels wide? That's... insane, especially for a WMS request (large jobs should be deferred to WPS asynch requests). Looking at the code, there is logic that generates the min and max zoom levels for the tile matrix, and it does something rather surprising... it picks the zoom level closest to the requested one as the min zoom level, and then starts adding zoom levels until the matrix has (stricly) more than 256 tiles in width... in my case, that means 512 tiles, or a total width of 512*256=131072 pixels! Trying GDAL on the command line, translating from a geotiff to a geopackage, it makes what looks like a more sensible choice instead: it uses the requested zoom level as the _maximum_ one, and then adds overviews to it, in the cases I've tried, until a single tile is left (basically, a good output with overviews). With our case we can also generate raster geopackages out of vector layers, and creating the overviews might end up hitting badly setup styles displaying too much data, but still, that would be a problem also in random WMS requests, so I'm not too concerned... Also, by default, GDAL generates a gridset that's an exact match for the source data, meaning a custom one. That also makes sense to me, if someone wants something other than the BBOX/SRS requested there are format options to choose both the target gridset and min/max zoom levels. Given that the modules are unsupported and not working properly anyways, I'd rather fix things to follow GDAL's approach. Any objection? Cheers Andrea -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal
