Hi Ben Thanks for explaining - Petr and I are the maintainers of OSM2Vectortiles. I like projections (and also GeoPackage :-) ). But reprojection is solved here, since OGR/QGIS can transform on the fly. This thread is about Petr's question about a reader "for mapbox-like vector tiles stored in MBTiles or downloaded from a url...".
So I'd like to come back to the issue: 1. of a generic MVT decoder/enoder library (C++ or Python) => Blake? 2. and of a "feature provider" in an interactive environment (like QGIS) or a file/stream converter (like OGR) => anybody? :Stefan 2016-02-02 0:04 GMT+01:00 Stadin, Benjamin <[email protected]>: > Hi Stefan, > > A vector tile may contain data for several zoom levels. But what data is > shown at which zoom level is defined by the styling, not by the data > itself. It is handled on client side, by the web (or native) tile > renderer. > > The path is the same as with mercator tiles (zoomLevel/x/y): > https://example.com/17/65535/43602.mvt > > > The tile for zoom level 17 may contain features of class „building“, and > the style definition may contain a rule that filters type „building“ for > zoomLevels <= 18. > > ~Ben > > Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter > <[email protected] on behalf of [email protected]>: > >>That's good news! >>We'll be happy to add Python bindings to such a generic C++ library >>for Mapbox vector tiles. >>We have still some time (say few weeks?) left here to decide to >>contribute either to these bindings or to Mapzen's pure Python lib. >> >>One reason why I'm insisting on Python (bindings) - besides easier >>testing with QGIS - is the following: >>AFAIK with vector tiles we have to deal with quite a new kind of >>feature provider: >>One which contains "zoom levels" and one which "suddenly" changes >>feature class at different zoom levels. >>If a client like QGIS consumes vector tiles from OGR in a one-shot >>call, how should the client deal with such "zoom level" dependent >>issues? >>So my thinking is, to consume only the most appropriate zoom level, >>probably only within current map canvas... >> >>:Stefan >> >>2016-02-01 18:27 GMT+01:00 Flippmoke <[email protected]>: >>> >>> Stefan, >>> >>> I am not aware of any libraries besides mapnik vector tile that have >>>been attempting to implement v2 specification. I think mapzen's >>>implementation might be best? We have been wanting to add vector tiles >>>to the python mapnik bindings for quite some time but we don't have the >>>time to do it right now. >>> >>> I will try to think this week about how we could make a generic C++ >>>library quickly for Mapbox vector tiles. >>> >>> Thanks, >>> >>> Blake Thompson >>> >>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[email protected]> wrote: >>>> >>>> Hi Blake >>>> >>>> It's clear to me that a good generic c++ library for encoding and >>>> decoding vector tiles is needed. >>>> On the other hand, I'd like to experiment with vector tiles and >>>> related QGIS parallel loading issues for which Python is more >>>> convenient. >>>> >>>> It seems that Mapzen's [1] Python implementation is further along. >>>> Or do you know if Jesse (or somebody) is still working on Mapbox's >>>>variant [2] ? >>>> >>>> :Stefan >>>> >>>> [1] https://github.com/mapzen/mapbox-vector-tile >>>> [2] https://github.com/mapbox/vector-tile-py >>>> >>>> >>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[email protected]>: >>>>> All, >>>>> >>>>> The first thing that is going to be required likely is a good generic >>>>>c++ library for encoding and decoding vector tiles IMO. >>>>> >>>>> I have spent a lot of time working on developing the mapnik vector >>>>>tile library to be a solid implementation, especially around all the >>>>>work done for 2.0 of Mapbox vector tile specification. I would love to >>>>>eventually help make this a generalized implementation so that it is >>>>>portable to a library like GDAL. >>>>> >>>>> We have talked about doing something like this at Mapbox but will >>>>>require a good deal of time, so it hasn't been pushed forward yet. The >>>>>other issues that come to mind to me is that some of the libraries >>>>>that mapnik vector tile depends upon most likely need good custom >>>>>implementations if we wanted to make it more generic. >>>>> >>>>> The reason for this is that I would prefer for it to be a header only >>>>>library with no dependencies. Adding more dependencies for GDAL seems >>>>>like it could be a mess. >>>>> >>>>> Blake Thompson >>>>> >>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[email protected]> >>>>>>wrote: >>>>>> >>>>>> Hi Petr, Blake, Even and everybody >>>>>> >>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles. >>>>>> I'd already asked Even a similar question. >>>>>> I'm advising now an intern to implement such a MVT Vector Tiles >>>>>>reader >>>>>> client in Python if possible as part of a QGIS plugin. >>>>>> And I'm interested not to do redundant implementations. >>>>>> Are there any news on this matter? >>>>>> >>>>>> :Stefan >>>>>> Prof. at HSR and leader of Geometa Lab >>>>>> >>>>>> @Even: Regarding OGR MVT development can you send to me (directly) a >>>>>> contact name of the group which won the bidding? >>>>>> >>>>>> >>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <[email protected]>: >>>>>>> Hi everybody, >>>>>>> >>>>>>> thanks for the comments. Regarding the technical details: >>>>>>> >>>>>>>> Each Tile is its own discrete dataset therefore the hardest part >>>>>>>>would be >>>>>>>> merging of features into the original features across tiles. You >>>>>>>>could use >>>>>>>> some complicated merging technique, but that could be very >>>>>>>>expensive. >>>>>>> >>>>>>> >>>>>>> The driver could require "id" parameter, which specifies name of an >>>>>>> attribute with unique identifier on the features shared across the >>>>>>>tiles. >>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique >>>>>>>non-zero >>>>>>> "osm_id" in place. The first version of driver could read only >>>>>>>features with >>>>>>> such ID. The identifier does not have to be always filled, but >>>>>>>often it is, >>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap >>>>>>>- but >>>>>>> yeh, a completely general stitching would be much harder. >>>>>>> >>>>>>> The clipping + merging of shapes itself can be implemented with >>>>>>>underlaying >>>>>>> GEOS library functions. >>>>>>> >>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript >>>>>>>X-Ray >>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3: >>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html >>>>>>> >>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL >>>>>>>JS >>>>>>> powered X-Ray: >>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html >>>>>>> >>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure >>>>>>> JavaScript: >>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html >>>>>>> >>>>>>> The OGR driver could be also practical for the unsimplified / >>>>>>>ungeneralised >>>>>>> MBTiles with OSM data available at: >>>>>>>http://osmlab.github.io/osm-qa-tiles/ >>>>>>> >>>>>>>> >>>>>>>> If you wanted to simply view vector tile data it could get >>>>>>>>confusing as >>>>>>>> well since most vector tiles need a buffer when they are used for >>>>>>>> visualization programs. This is important for things like labeling >>>>>>>>on maps >>>>>>>> etc. Therefore, you could get a large amount of overlapping data >>>>>>>>on tiles if >>>>>>>> you loaded them all in at once. >>>>>>> >>>>>>> >>>>>>> Clipping on border of each tile must be applied before stitching >>>>>>>the vector >>>>>>> features together - so the buffer is removed. Buffer is there only >>>>>>>for >>>>>>> visualisation and labeling. >>>>>>> >>>>>>>> >>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile >>>>>>>>decoder isn't >>>>>>>> currently able to decode vector tiles into a form not supported by >>>>>>>>mapnik. >>>>>>>> Therefore, OGR would not be able to easily use this. >>>>>>> >>>>>>> >>>>>>> Correct. It can't be used directly (GDAL can't link against), but >>>>>>>the >>>>>>> relevant code can be extracted and reused directly. For basic >>>>>>>decoding >>>>>>> critical is the Protobuf library (which is already in GDAL) and >>>>>>> >>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector >>>>>>>_tile.proto >>>>>>> + the code for converting pixel coordinates in features to relevant >>>>>>>geo >>>>>>> coordinates and GEOS-based merging of features. >>>>>>> >>>>>>>> We plan to write a generalized encoder and decoder eventually so >>>>>>>>that any >>>>>>>> other library can plugin with out the mapnik requirement. >>>>>>> >>>>>>> >>>>>>> This would be amazing. >>>>>>> >>>>>>>> If you have any questions specifically about Mapnik Vector Tile or >>>>>>>>the >>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note >>>>>>>>that you >>>>>>>> guys will want to update your vector tiles once the V2 >>>>>>>>specification changes >>>>>>>> are into Mapnik-Vector-Tile. >>>>>>> >>>>>>> >>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers >>>>>>>on a >>>>>>> cluster of computers - using Mapnik via Tilelive directly - so >>>>>>>upgrades on >>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering >>>>>>>software. >>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right? >>>>>>> >>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I >>>>>>>>don't >>>>>>>> see >>>>>>>> mention of that neither in the MVT spec ( >>>>>>>> https://github.com/mapbox/vector- >>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one ( >>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), >>>>>>>>although >>>>>>>> I >>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data >>>>>>>>column. >>>>>>> >>>>>>> >>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles >>>>>>>inside - >>>>>>> and Mapnik reads them directly when rasterising. Storing the vector >>>>>>>tiles in >>>>>>> SQLite make sense - and it simplifies deployment, and you can then >>>>>>>easily >>>>>>> have complete world on each node of a cluster of tileservers. If >>>>>>>you don't >>>>>>> want to upgrade your base map too often, this approach is pretty >>>>>>>efficient >>>>>>> and removes the centralised database as typical bottleneck. >>>>>>> >>>>>>> I did some tests on vector tiles vs raster in MBTiles and >>>>>>>documented these >>>>>>> in README.md at and >>>>>>>https://github.com/klokantech/vector-tiles-sample before >>>>>>> we started to work on OSM2VectorTiles project. >>>>>>> There are also sample data and offline MapBox GL JS viewer in that >>>>>>>repo. >>>>>>> Vector tiles can be hosted also "unpacked", just as files in >>>>>>>folders on a >>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does >>>>>>>by >>>>>>> default for raster tiles. >>>>>>> >>>>>>> If OGR driver is implemented, the primary source should be probably >>>>>>>reading >>>>>>> from an URL via curl. >>>>>>> >>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most >>>>>>>practical >>>>>>> portable alternative to it. >>>>>>> >>>>>>> Technically people may store the tiles in different ways - for >>>>>>>example >>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and >>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I >>>>>>>guess >>>>>>> MapBox internally is also on a different storage for production >>>>>>>servers ... >>>>>>> for transfers and streaming tilesets there is the - but tiles are >>>>>>>ALWAYS >>>>>>> exposed via HTTP. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Petr >>>>>>> -- >>>>>>> Petr Pridal, Ph.D. >>>>>>> CEO >>>>>>> >>>>>>> Klokan Technologies GmbH >>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland >>>>>>> Tel: +41 (0)41 511 26 12 >>>>>>> Email: [email protected] >>>>>>> Web: http://www.klokantech.com/ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> gdal-dev mailing list >>>>>>> [email protected] >>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev >>_______________________________________________ >>gdal-dev mailing list >>[email protected] >>http://lists.osgeo.org/mailman/listinfo/gdal-dev > _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
