Blake, Thanks for the comments. Regarding leaving out data at intermediate zoom levels I also meant omitting (setting to null) the entire tile. In fact I just realize that the spec. even does not mention how a reader knows at which level it should expect data if a tile is null/empty.
:Stefan 2016-02-02 2:18 GMT+01:00 Blake Thompson <[email protected]>: > Stefan, > > Consistent recognition of specific features can be done by proper use of the > id field in features: > > https://github.com/mapbox/vector-tile-spec/tree/master/2.1#42-features > > You can definitely leave out data at intermediate zooms if you wished, we > have considered this on many occasions at Mapbox. As far as the > specification is concerned currently, the vector tile itself purposefully > does not contain all the metadata necessary for this specific task. This is > mentioned directly in the specification that each tile does not contain > information about its projection or bounds. The Vector Tile Specification is > purposeful in this because it leaves others specifications or an > implementation to explicitly provide all the necessary information. > > I have considered that we could define a metafile to define all this > information explicitly, but for now as far as I am concerned the > specification just for the tiles is very young! We could make a decision on > this sort of implementation, but it could very well be better placed inside > another standard. I will not shy away from the fact that the specification > very well might change as time goes on and perhaps adding more things that > will more narrowly define it such things as projection and bounds. The needs > of those using the format will definitely provide influence on how it > changes. > > There currently is no indexing or listing of features within a layer, other > then the layer classes themselves. We could provide such a feature perhaps, > but currently it would likely just take more memory with out much benefit. > It is very quick for our decoder to iterate over features, and in general > processing of vector tile data is expected to be done in memory. > > Most client are only concerned about the vector tiles that are currently > within a "view". Therefore, if a feature is missing within a view it is > considered not to have that feature. However, this doesn't limit people from > doing analysis across a large number of tiles. At Mapbox we have used > map-reduce type solutions to process large number of tiles for different > projects: > > https://github.com/mapbox/tile-reduce > > The idea is that vector tiles themselves should be relatively small in size > in most situations. If you data becomes too dense you could put the data at > a higher zoom or provide an abstraction of features. The format itself is a > "lossy" format as you are putting your data in a projection that is unique > to the tile, so the format is not designed for situations where accuracy is > extremely important. If you need more accuracy you can increase the "extent" > of a layer, which adds more precision, but this could also increase the size > of your tiles. > > As far as the concerns with making a solid GDAL driver from vector tiles, I > am not certain what the solution should be honestly. I want to be here as a > reference to help something be grown, but I can't point at a great > implementation like this because we have not put into mapnik even yet "a > datasource of many vector tiles" as a plugin. We have left such operations > up to a large number of open source javascript libraries that can provide > things like caching etc. (See tilelive https://github.com/mapbox/tilelive/ ) > > In a limited sense to think of Vector Tiles for Mapbox Tools in the same > sense you would think about WKB for PostGIS. It is not the entire solution > alone. > > Thanks, > > Blake Thompson > > On Mon, Feb 1, 2016 at 5:45 PM, Stefan Keller <[email protected]> wrote: >> >> Blake, >> >> Now the vector tiles spec. and the Mapbox approach became clearer to me. >> >> I'm still pondering about realizing a vector tiles reader (especially >> for QGIS) and for vector tiles uses cases like 1. base maps and 2. >> spatial analysis: >> >> Regarding clipped linestrings and polygons arriving at reader, there >> seems to exist no hint (clipped) nor id mentioned in the spec. to help >> the reader merging them? >> >> While there's overzoom, what's about leaving out data in intermediate >> zoom levels to save space? Does the spec. regulate that and who's >> responsible for handling this: server or client? >> >> Finally and back to my implementation concerns: How should an >> interactive client deal occurring/disappearing feature classes in a >> single "layer"? Is it forced to read all features on all zoom levels >> in order to announce all feature classes? >> >> :Stefan >> >> >> 2016-02-02 1:03 GMT+01:00 Blake Thompson <[email protected]>: >> > Benjamin Stadin, >> > >> > As an author of the specification, I have to disagree with you on the >> > support for other projections. There is no limitation on vector tiles >> > being >> > required to being in mercator. The specification in fact mentions that >> > other >> > projections may be used: >> > >> > >> > https://github.com/mapbox/vector-tile-spec/tree/master/2.1#3-projection-and-bounds >> > >> > While it is not accessible via the node-mapnik bindings, we do in fact >> > support encoding as different projections in mapnik-vector-tile. We have >> > implemented the decoder so that it could accept other projections, but >> > have >> > not added a mapnik datasource that would support them, but if there is >> > demand for such a thing we could do it. >> > >> > Your comments about the entire stack of Mapbox supporting mostly >> > mercator >> > could be very accurate, but this is not at all related to the Vector >> > Tile >> > Specification. We purposefully attempt to make our tools as >> > projection-less >> > as possible and hope that others find awesome ways to build upon the >> > standard. >> > >> > "But what data is shown at which zoom level is defined by the styling" >> > -- >> > This is not true because you could make your data not even provide data >> > at >> > certain zoom levels. At Mapbox we do this quite often, street data is >> > never >> > included at very low zoom levels. Therefore, the style isn't the only >> > control here. In fact, I would encourage you to view styling and >> > rendering >> > of vector tiles as an entirely different topic. >> > >> > "There is enough interest from our side that we¹d be interested in >> > contributing (maybe monetary or with code, or both) to define and >> > implement >> > a better general purpose tile format." -- Any issues you might have with >> > the >> > Mapbox Vector Tile Specification or any ideas you might have with the >> > format >> > are very much welcome. We want to make sure the spec migrates for >> > community >> > needs. Please see >> > https://github.com/mapbox/vector-tile-spec/blob/master/CONTRIBUTING.md >> > for >> > how you might help! >> > >> > Finally, I would like to stress that while Mapbox Vector Tiles were >> > designed >> > to be for rendering, it was well known during their design that they >> > could >> > be used for other applications. This was especially true when I was >> > writing >> > 2.0 of the specification as I tried to limit what would be considered >> > even >> > the most common uses of vector tiles. >> > >> > Thanks, >> > >> > Blake Thompson >> > >> > On Mon, Feb 1, 2016 at 4:27 PM, Stefan Keller <[email protected]> >> > wrote: >> >> >> >> Hi Ben >> >> >> >> Thanks for explaining - Petr and I are the maintainers of >> >> OSM2Vectortiles. >> >> I like projections (and also GeoPackage :-) ). >> >> But reprojection is solved here, since OGR/QGIS can transform on the >> >> fly. >> >> This thread is about Petr's question about a reader "for mapbox-like >> >> vector tiles stored in MBTiles or downloaded from a url...". >> >> >> >> So I'd like to come back to the issue: >> >> 1. of a generic MVT decoder/enoder library (C++ or Python) => Blake? >> >> 2. and of a "feature provider" in an interactive environment (like >> >> QGIS) or a file/stream converter (like OGR) => anybody? >> >> >> >> :Stefan >> >> >> >> >> >> >> >> 2016-02-02 0:04 GMT+01:00 Stadin, Benjamin >> >> <[email protected]>: >> >> > Hi Stefan, >> >> > >> >> > A vector tile may contain data for several zoom levels. But what data >> >> > is >> >> > shown at which zoom level is defined by the styling, not by the data >> >> > itself. It is handled on client side, by the web (or native) tile >> >> > renderer. >> >> > >> >> > The path is the same as with mercator tiles (zoomLevel/x/y): >> >> > https://example.com/17/65535/43602.mvt >> >> > >> >> > >> >> > The tile for zoom level 17 may contain features of class „building“, >> >> > and >> >> > the style definition may contain a rule that filters type „building“ >> >> > for >> >> > zoomLevels <= 18. >> >> > >> >> > ~Ben >> >> > >> >> > Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" >> >> > unter >> >> > <[email protected] on behalf of [email protected]>: >> >> > >> >> >>That's good news! >> >> >>We'll be happy to add Python bindings to such a generic C++ library >> >> >>for Mapbox vector tiles. >> >> >>We have still some time (say few weeks?) left here to decide to >> >> >>contribute either to these bindings or to Mapzen's pure Python lib. >> >> >> >> >> >>One reason why I'm insisting on Python (bindings) - besides easier >> >> >>testing with QGIS - is the following: >> >> >>AFAIK with vector tiles we have to deal with quite a new kind of >> >> >>feature provider: >> >> >>One which contains "zoom levels" and one which "suddenly" changes >> >> >>feature class at different zoom levels. >> >> >>If a client like QGIS consumes vector tiles from OGR in a one-shot >> >> >>call, how should the client deal with such "zoom level" dependent >> >> >>issues? >> >> >>So my thinking is, to consume only the most appropriate zoom level, >> >> >>probably only within current map canvas... >> >> >> >> >> >>:Stefan >> >> >> >> >> >>2016-02-01 18:27 GMT+01:00 Flippmoke <[email protected]>: >> >> >>> >> >> >>> Stefan, >> >> >>> >> >> >>> I am not aware of any libraries besides mapnik vector tile that >> >> >>> have >> >> >>>been attempting to implement v2 specification. I think mapzen's >> >> >>>implementation might be best? We have been wanting to add vector >> >> >>> tiles >> >> >>>to the python mapnik bindings for quite some time but we don't have >> >> >>> the >> >> >>>time to do it right now. >> >> >>> >> >> >>> I will try to think this week about how we could make a generic C++ >> >> >>>library quickly for Mapbox vector tiles. >> >> >>> >> >> >>> Thanks, >> >> >>> >> >> >>> Blake Thompson >> >> >>> >> >> >>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <[email protected]> >> >> >>>> wrote: >> >> >>>> >> >> >>>> Hi Blake >> >> >>>> >> >> >>>> It's clear to me that a good generic c++ library for encoding and >> >> >>>> decoding vector tiles is needed. >> >> >>>> On the other hand, I'd like to experiment with vector tiles and >> >> >>>> related QGIS parallel loading issues for which Python is more >> >> >>>> convenient. >> >> >>>> >> >> >>>> It seems that Mapzen's [1] Python implementation is further along. >> >> >>>> Or do you know if Jesse (or somebody) is still working on Mapbox's >> >> >>>>variant [2] ? >> >> >>>> >> >> >>>> :Stefan >> >> >>>> >> >> >>>> [1] https://github.com/mapzen/mapbox-vector-tile >> >> >>>> [2] https://github.com/mapbox/vector-tile-py >> >> >>>> >> >> >>>> >> >> >>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <[email protected]>: >> >> >>>>> All, >> >> >>>>> >> >> >>>>> The first thing that is going to be required likely is a good >> >> >>>>> generic >> >> >>>>>c++ library for encoding and decoding vector tiles IMO. >> >> >>>>> >> >> >>>>> I have spent a lot of time working on developing the mapnik >> >> >>>>> vector >> >> >>>>>tile library to be a solid implementation, especially around all >> >> >>>>> the >> >> >>>>>work done for 2.0 of Mapbox vector tile specification. I would >> >> >>>>> love >> >> >>>>> to >> >> >>>>>eventually help make this a generalized implementation so that it >> >> >>>>> is >> >> >>>>>portable to a library like GDAL. >> >> >>>>> >> >> >>>>> We have talked about doing something like this at Mapbox but will >> >> >>>>>require a good deal of time, so it hasn't been pushed forward yet. >> >> >>>>> The >> >> >>>>>other issues that come to mind to me is that some of the libraries >> >> >>>>>that mapnik vector tile depends upon most likely need good custom >> >> >>>>>implementations if we wanted to make it more generic. >> >> >>>>> >> >> >>>>> The reason for this is that I would prefer for it to be a header >> >> >>>>> only >> >> >>>>>library with no dependencies. Adding more dependencies for GDAL >> >> >>>>> seems >> >> >>>>>like it could be a mess. >> >> >>>>> >> >> >>>>> Blake Thompson >> >> >>>>> >> >> >>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <[email protected]> >> >> >>>>>>wrote: >> >> >>>>>> >> >> >>>>>> Hi Petr, Blake, Even and everybody >> >> >>>>>> >> >> >>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles. >> >> >>>>>> I'd already asked Even a similar question. >> >> >>>>>> I'm advising now an intern to implement such a MVT Vector Tiles >> >> >>>>>>reader >> >> >>>>>> client in Python if possible as part of a QGIS plugin. >> >> >>>>>> And I'm interested not to do redundant implementations. >> >> >>>>>> Are there any news on this matter? >> >> >>>>>> >> >> >>>>>> :Stefan >> >> >>>>>> Prof. at HSR and leader of Geometa Lab >> >> >>>>>> >> >> >>>>>> @Even: Regarding OGR MVT development can you send to me >> >> >>>>>> (directly) >> >> >>>>>> a >> >> >>>>>> contact name of the group which won the bidding? >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal >> >> >>>>>> <[email protected]>: >> >> >>>>>>> Hi everybody, >> >> >>>>>>> >> >> >>>>>>> thanks for the comments. Regarding the technical details: >> >> >>>>>>> >> >> >>>>>>>> Each Tile is its own discrete dataset therefore the hardest >> >> >>>>>>>> part >> >> >>>>>>>>would be >> >> >>>>>>>> merging of features into the original features across tiles. >> >> >>>>>>>> You >> >> >>>>>>>>could use >> >> >>>>>>>> some complicated merging technique, but that could be very >> >> >>>>>>>>expensive. >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> The driver could require "id" parameter, which specifies name >> >> >>>>>>> of >> >> >>>>>>> an >> >> >>>>>>> attribute with unique identifier on the features shared across >> >> >>>>>>> the >> >> >>>>>>>tiles. >> >> >>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an >> >> >>>>>>> unique >> >> >>>>>>>non-zero >> >> >>>>>>> "osm_id" in place. The first version of driver could read only >> >> >>>>>>>features with >> >> >>>>>>> such ID. The identifier does not have to be always filled, but >> >> >>>>>>>often it is, >> >> >>>>>>> especially in deeper zoom levels derived purely from >> >> >>>>>>> OpenStreetMap >> >> >>>>>>>- but >> >> >>>>>>> yeh, a completely general stitching would be much harder. >> >> >>>>>>> >> >> >>>>>>> The clipping + merging of shapes itself can be implemented with >> >> >>>>>>>underlaying >> >> >>>>>>> GEOS library functions. >> >> >>>>>>> >> >> >>>>>>> BTW It is possible to examine vector tiles data in pure >> >> >>>>>>> JavaScript >> >> >>>>>>>X-Ray >> >> >>>>>>> viewer (no WebGL support on browser required) made in >> >> >>>>>>> OpenLayers3: >> >> >>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html >> >> >>>>>>> >> >> >>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox >> >> >>>>>>> GL >> >> >>>>>>>JS >> >> >>>>>>> powered X-Ray: >> >> >>>>>>> >> >> >>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html >> >> >>>>>>> >> >> >>>>>>> A debug viewer for each vector tile (made in OL3) decoded in >> >> >>>>>>> pure >> >> >>>>>>> JavaScript: >> >> >>>>>>> >> >> >>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html >> >> >>>>>>> >> >> >>>>>>> The OGR driver could be also practical for the unsimplified / >> >> >>>>>>>ungeneralised >> >> >>>>>>> MBTiles with OSM data available at: >> >> >>>>>>>http://osmlab.github.io/osm-qa-tiles/ >> >> >>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> If you wanted to simply view vector tile data it could get >> >> >>>>>>>>confusing as >> >> >>>>>>>> well since most vector tiles need a buffer when they are used >> >> >>>>>>>> for >> >> >>>>>>>> visualization programs. This is important for things like >> >> >>>>>>>> labeling >> >> >>>>>>>>on maps >> >> >>>>>>>> etc. Therefore, you could get a large amount of overlapping >> >> >>>>>>>> data >> >> >>>>>>>>on tiles if >> >> >>>>>>>> you loaded them all in at once. >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> Clipping on border of each tile must be applied before >> >> >>>>>>> stitching >> >> >>>>>>>the vector >> >> >>>>>>> features together - so the buffer is removed. Buffer is there >> >> >>>>>>> only >> >> >>>>>>>for >> >> >>>>>>> visualisation and labeling. >> >> >>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile >> >> >>>>>>>>decoder isn't >> >> >>>>>>>> currently able to decode vector tiles into a form not >> >> >>>>>>>> supported >> >> >>>>>>>> by >> >> >>>>>>>>mapnik. >> >> >>>>>>>> Therefore, OGR would not be able to easily use this. >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> Correct. It can't be used directly (GDAL can't link against), >> >> >>>>>>> but >> >> >>>>>>>the >> >> >>>>>>> relevant code can be extracted and reused directly. For basic >> >> >>>>>>>decoding >> >> >>>>>>> critical is the Protobuf library (which is already in GDAL) and >> >> >>>>>>> >> >> >> >> >>>>>>> >> >> >>>>>>> >>>>>>> >>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector >> >> >>>>>>>_tile.proto >> >> >>>>>>> + the code for converting pixel coordinates in features to >> >> >>>>>>> relevant >> >> >>>>>>>geo >> >> >>>>>>> coordinates and GEOS-based merging of features. >> >> >>>>>>> >> >> >>>>>>>> We plan to write a generalized encoder and decoder eventually >> >> >>>>>>>> so >> >> >>>>>>>>that any >> >> >>>>>>>> other library can plugin with out the mapnik requirement. >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> This would be amazing. >> >> >>>>>>> >> >> >>>>>>>> If you have any questions specifically about Mapnik Vector >> >> >>>>>>>> Tile >> >> >>>>>>>> or >> >> >>>>>>>>the >> >> >>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would >> >> >>>>>>>> note >> >> >>>>>>>>that you >> >> >>>>>>>> guys will want to update your vector tiles once the V2 >> >> >>>>>>>>specification changes >> >> >>>>>>>> are into Mapnik-Vector-Tile. >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> Cool! Thanks. Our world tiles are generated via Docker >> >> >>>>>>> containers >> >> >>>>>>>on a >> >> >>>>>>> cluster of computers - using Mapnik via Tilelive directly - so >> >> >>>>>>>upgrades on >> >> >>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering >> >> >>>>>>>software. >> >> >>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right? >> >> >>>>>>> >> >> >>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice >> >> >>>>>>>> ? I >> >> >>>>>>>>don't >> >> >>>>>>>> see >> >> >>>>>>>> mention of that neither in the MVT spec ( >> >> >>>>>>>> https://github.com/mapbox/vector- >> >> >>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one ( >> >> >>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md >> >> >>>>>>>> ), >> >> >>>>>>>>although >> >> >>>>>>>> I >> >> >>>>>>>> guess it is just a matter of storing a MVT blob in the >> >> >>>>>>>> tile_data >> >> >>>>>>>>column. >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector >> >> >>>>>>> Tiles >> >> >>>>>>>inside - >> >> >>>>>>> and Mapnik reads them directly when rasterising. Storing the >> >> >>>>>>> vector >> >> >>>>>>>tiles in >> >> >>>>>>> SQLite make sense - and it simplifies deployment, and you can >> >> >>>>>>> then >> >> >>>>>>>easily >> >> >>>>>>> have complete world on each node of a cluster of tileservers. >> >> >>>>>>> If >> >> >>>>>>>you don't >> >> >>>>>>> want to upgrade your base map too often, this approach is >> >> >>>>>>> pretty >> >> >>>>>>>efficient >> >> >>>>>>> and removes the centralised database as typical bottleneck. >> >> >>>>>>> >> >> >>>>>>> I did some tests on vector tiles vs raster in MBTiles and >> >> >>>>>>>documented these >> >> >>>>>>> in README.md at and >> >> >>>>>>>https://github.com/klokantech/vector-tiles-sample before >> >> >>>>>>> we started to work on OSM2VectorTiles project. >> >> >>>>>>> There are also sample data and offline MapBox GL JS viewer in >> >> >>>>>>> that >> >> >>>>>>>repo. >> >> >>>>>>> Vector tiles can be hosted also "unpacked", just as files in >> >> >>>>>>>folders on a >> >> >>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com >> >> >>>>>>> does >> >> >>>>>>>by >> >> >>>>>>> default for raster tiles. >> >> >>>>>>> >> >> >>>>>>> If OGR driver is implemented, the primary source should be >> >> >>>>>>> probably >> >> >>>>>>>reading >> >> >>>>>>> from an URL via curl. >> >> >>>>>>> >> >> >>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the >> >> >>>>>>> most >> >> >>>>>>>practical >> >> >>>>>>> portable alternative to it. >> >> >>>>>>> >> >> >>>>>>> Technically people may store the tiles in different ways - for >> >> >>>>>>>example >> >> >>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and >> >> >>>>>>> https://github.com/kartotherian/) use Cassandra for tile >> >> >>>>>>> storage, >> >> >>>>>>> I >> >> >>>>>>>guess >> >> >>>>>>> MapBox internally is also on a different storage for production >> >> >>>>>>>servers ... >> >> >>>>>>> for transfers and streaming tilesets there is the - but tiles >> >> >>>>>>> are >> >> >>>>>>>ALWAYS >> >> >>>>>>> exposed via HTTP. >> >> >>>>>>> >> >> >>>>>>> Regards, >> >> >>>>>>> >> >> >>>>>>> Petr >> >> >>>>>>> -- >> >> >>>>>>> Petr Pridal, Ph.D. >> >> >>>>>>> CEO >> >> >>>>>>> >> >> >>>>>>> Klokan Technologies GmbH >> >> >>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland >> >> >>>>>>> Tel: +41 (0)41 511 26 12 >> >> >>>>>>> Email: [email protected] >> >> >>>>>>> Web: http://www.klokantech.com/ >> >> >>>>>>> >> >> >>>>>>> _______________________________________________ >> >> >>>>>>> gdal-dev mailing list >> >> >>>>>>> [email protected] >> >> >>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev >> >> >>_______________________________________________ >> >> >>gdal-dev mailing list >> >> >>[email protected] >> >> >>http://lists.osgeo.org/mailman/listinfo/gdal-dev >> >> > >> > >> > > > _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
