Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
As Mapbox GL has been un-open sourced, I'm putting this project on hold until I figure out a suitable rendering engine. See https://github.com/pnorman/openstreetmap-cartographic/issues/7 On 2020-05-24 5:34 p.m., Paul Norman via dev wrote: I've been working on a new project, OpenStreetMap Cartographic. This is a client-side rendering based on OpenStreetMap Carto. This is an ambitious project, as OpenStreetMap Carto is an extremely complex style which shows a large number of features. The technical choices I'm making are designed so the style is capable of handling the load of osm.org with minutely updates. I've put up a world-wide demo at https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html, using data from 2020-03-16, and you can view the code at https://github.com/pnorman/openstreetmap-cartographic. Incomplete parts Only zoom 0 to 8 has been implemented so far. I started at zoom 0 and am working my way down. Admin boundaries are not implemented. OpenStreetMap Carto uses Mapnik-specific tricks to deduplicate the rendering of these. I know how I can do this, but it requires the changes I intend to make with the flex backend. Landuse, vegetation, and other natural features are not rendered until zoom 7. This is the scale of OpenStreetMap Carto zoom 8, and these features first appear at zoom 5. There are numerous problems with unprocessed OpenStreetMap data at these scales. OpenStreetMap Carto gets a result that looks acceptable but is poor at conveying information by tweaking Mapnik image rasterizing options. I'm looking for better options here involving preprocessed data, but haven't found any. I'm still investigating how to best distribute sprites. Technology == The technology choices are designed to be suitable for a replacement for tile.osm.org. This means minutely updates, high traffic, high reliability, and multiple servers. Tilekiln, the vector tile generator, supports all of these. It's designed to better share the rendering results among multiple servers, a significant flaw with renderd + mod_tile and the standard filesystem storage. It uses PostGIS' ST_AsMVT, which is very fast with PostGIS 3.0. On my home system it generates z0-z8 in under 40 minutes. Often forgotten is the development requirements. The style needs to support multiple developers working on similar areas, git merge conflicts while maintaining an easy development workflow. I'm still figuring this out. Mapbox GL styles are written in JSON and most of the tools overwrite any formatting. This means there's no way to add comments to lines of codes. Comments are a requirement for a style like this, so I'm investigating minimal pre-processing options. The downside to this will make it harder to use with existing GUI editors like Fresco or Maputnik. Cartography === The goal of this project isn't to do big cartography changes yet, but client-side rendering opens up new tools. The biggest immediate change is zoom is continuous, no longer an integer or fixed value. This means parameters like sizes can smoothly change as you zoom in and out, specified by their start and end size instead of having to specify each zoom. Want to help? = Have a look at https://github.com/pnorman/openstreetmap-cartographic and have a go at setting it up and generating your own map. If you have issues, open an issue or pull request. Or, because OpenStreetMap Cartographic uses Tilekiln have a look at https://github.com/pnorman/tilekiln/issues. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
Not sure I understand enough well the many technologies you want to mix here, especially the essential once like tilekiln, vectortiles, CartoCSS stylesheet … where you are the author/maintainer and where “… not provided a project description”. Yet, I do understand, you intend to develop a vector tiles based server-(mobile)client cartographic service using the OSM source data. Being there in a really long time and having developed several innovative related technologies, I will mention some traps and issues you will meet on your way “...at zoom 0 and am working my way down”. Being aware of them you may save lot of time and effort. Besides, some of the suggestions might help you to really make a difference. If you jump over the data preparation (you may call it pre-processing) initial phase of the development, you will not reach further than the existing OSM maps. Lots of the data anomalies in the source will inevitably limit the quality and the efficiency of your further work on scale levels (vs zoom levels) generation, data generalisation, vector tiling, rendering on the client side, and so on. Among huge number of these let me mention just a few: object fragmentation, object replication (exact, almost, part of it...), missing parts or objects (fragments), unsynchronized objects in different classes, spiky start-end line connections, illogical/meaningless objects and object presentations, and so on. Without a robust preparation the consequences may be quite serious both visually and functionally (even leading to useless maps), especially in the vector map/making. Of cause, you/anyone working with vector map-making, may find large number of examples to illustrate the mentioned anomalies and the consequences. A robust preparation process may remove/correct most of the (well over hundred thousands) mentioned anomalies. The next phase, the server data preparation you will probably start with the scale levels generation and end with the pre-tiling process. Besides the point object types, you have probably selected some 80 to 120 area and line modelled types that provide input to a very reach base-map. The scale levels numbering and the scale factor values should definitely not follow those defined by the OSM raster zoom levels (though, formally they can). Consider the data size criteria after scaling to a scale level. To (radically) reduce the data size in the level you will inevitably want to apply a cartographic generalisation. Consider a model that never causes fragmentation/brakes on solid objects (one of the worst consequences). The process should remove very small objects and tiny sections and perform vector smoothing (reduction). Consider dynamic criterion based smoothing (on fine details short, while on smooth details long vectors). Here you may want to do corrections on parallel roads, self-crossing polygons, inner-outer polygon crossings and so on. But this depends whether you use an old fashion or a modern topology base rendering algorithm on the client side.If the object class scale level is not empty, you would probably run a tiling process. If you want an attractive arbitrary (vs. “continuous”) scaling and rotation, object layers switch on/of, z-moving, transparency, anchoring, panning… consider pre-tiling per object layers instead of all object types per tile. If you want a really innovative vector server-client system consider a multi-tiling based on-the-fly tiling where the final tiles to transmit are created according to the individual client request parameters. This model might provide unparalleled efficiency and flexibility. There is no need for extended tiles, unclipped objects, caching (yet, clients may have individual strategies) and so on. On the client side the only thing should remain to do is the anti-alias rendering. As mentioned, if on-the-fly multi-tiling is used, everything is done on the server except the rendering. Besides the default stiles, consider allowing users to define arbitrary stiles. For the line objects, you may find highly efficient thick line renderers (you know the thick line paradigm) where the border lines, stilling… you get for free and the speed is practically the same as for the thin (one pixel) lines. For the area fill, consider a topology (inner, border and outer points) based algorithm. As a role, such models tolerate self-closings, touching… even inner polygons crossing outers. Again, the arbitrary fill styling you get with these for free. Finally, I warmly suggest considering use of image based fonts for labelling/texts instead of classical (built in) fonts. Though, you can not render large characters, the size of an image font is hundreds of times smaller compared to the built in vector font(s). From the same source you can create arbitrary character sets and you have simple and full control of all character and text parameters. You can even have a dynamic labelling (e.g. the street name is moving to the left
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
Not sure I understand enough well the many technologies you want to mix here, especially the essential once like tilekiln, vectortiles, CartoCSS stylesheet … where you are the author/maintainer and where “… not provided a project description”. Yet, I do understand, you intend to develop a vector tiles based server-(mobile)client cartographic service using the OSM source data. Being there in a really long time and having developed several innovative related technologies, I will mention some traps and issues you will meet on your way “...at zoom 0 and am working my way down”. Being aware of them you may save lot of time and effort. Besides, some of the suggestions might help you to really make a difference.If you jump over the data preparation (you may call it pre-processing) initial phase of the development, you will not reach further than the existing OSM maps. Lots of the data anomalies in the source will inevitably limit the quality and the efficiency of your further work on scale levels (vs zoom levels) generation, data generalisation, vector tiling, rendering on the client side, and so on. Among huge number of these let me mention just a few: object fragmentation, object replication (exact, almost, part of it...), missing parts or objects (fragments), unsynchronized objects in different classes, spiky start-end line connections, illogical/meaningless objects and object presentations, and so on. Without a robust preparation the consequences may be quite serious both visually and functionally (even leading to useless maps), especially in the vector map/making. Of cause, you/anyone working with vector map-making, may find large number of examples to illustrate the mentioned anomalies and the consequences. A robust preparation process may remove/correct most of the (well over hundred thousands) mentioned anomalies.The next phase, the server data preparation you will probably start with the scale levels generation and end with the pre-tiling process. Besides the point object types, you have probably selected some 80 to 120 area and line modelled types that provide input to a very reach base-map. The scale levels numbering and the scale factor values should definitely not follow those defined by the OSM raster zoom levels (though, formally they can). Consider the data size criteria after scaling to a scale level. To (radically) reduce the data size in the level you will inevitably want to apply a cartographic generalisation. Consider a model that never causes fragmentation/brakes on solid objects (one of the worst consequences). The process should remove very small objects and tiny sections and perform vector smoothing (reduction). Consider dynamic criterion based smoothing (on fine details short, while on smooth details long vectors). Here you may want to do corrections on parallel roads, self-crossing polygons, inner-outer polygon crossings and so on. But this depends whether you use an old fashion or a modern topology base rendering algorithm on the client side.If the object class scale level is not empty, you would probably run a tiling process. If you want an attractive arbitrary (vs. “continuous”) scaling and rotation, object layers switch on/of, z-moving, transparency, anchoring, panning… consider pre-tiling per object layers instead of all object types per tile. If you want a really innovative vector server-client system consider a multi-tiling based on-the-fly tiling where the final tiles to transmit are created according to the individual client request parameters. This model might provide unparalleled efficiency and flexibility. There is no need for extended tiles, unclipped objects, caching (yet, clients may have individual strategies) and so on.On the client side the only thing should remain to do is the anti-alias rendering. As mentioned, if on-the-fly multi-tiling is used, everything is done on the server except the rendering. Besides the default stiles, consider allowing users to define arbitrary stiles. For the line objects, you may find highly efficient thick line renderers (you know, the thick line paradigm) where the border lines, stilling… you get for free and the speed is practically the same as for the thin (one pixel) lines. For the area fill, consider a topology (inner, border and outer points) based algorithm. As a role, such models tolerate self-closings, touching… even inner polygons crossing outers. Again, the arbitrary fill styling you get with these for free. Finally, I warmly suggest considering use of image based fonts for labelling/texts instead of classical (built in) fonts. Though, you can not render large characters, the size of an image font is hundreds of times smaller compared to the built in vector font(s). From the same source you can create arbitrary character sets and you have simple and full control of all character and text parameters. You can even have a dynamic labelling (e.g. the street name is moving to the left when panning to the right). There are huge
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
I just wanted to thank you for working on this. (Sadly I have no useful comments, hopefully such message is ok) 11 Jun 2020, 05:32 by dev@openstreetmap.org: > I've been busy with > > On 2020-05-24 10:26 p.m., Joseph Eisenberg wrote: > >> Thank you for making this, it looks like a lot of work! >> >> These client-side vector tiles at z8? >> (https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html) >> >> My laptop (2015 Macbook Pro, 16 GB RAM, 2.2 GHz Intel Core i7) appears to >> use some effort to show areas with a large amount of data, >> For example if I view England + Wales on z8 and then move over to the >> Netherlands and then to Germany, at z7 and z8, >> CPU usage goes up to >100% for a time, and there is a noticeable delay. >> Perhaps part of this is a delay in serving the tiles? >> > > I turned on Content-Encoding based compression which cuts the bytes > transferred in half. This helps significantly. > > In general vector tiles have more large tiles, even if the average tile size > is the same. The largest z8 tile in the area is 1MB transfer size while an > average weighted by requests on the tile.osm.org level 2 cache is 189k > uncompressed. For comparison, four z9 raster tiles from tile.osm.org are > about 200kb, a fifth of the bytes transferred. Retina tiles would decrease > the size different. > > > > > ___ > dev mailing list > dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/dev >___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
Hello, > On 2020-05-24 10:26 p.m., Joseph Eisenberg wrote: > > Thank you for making this, it looks like a lot of work! > > > > These client-side vector tiles at z8? > > (https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html) > > > > I turned on Content-Encoding based compression which cuts the bytes > transferred in half. This helps significantly. > > In general vector tiles have more large tiles, even if the average tile > size is the same. The largest z8 tile in the area is 1MB transfer size > while an average weighted by requests on the tile.osm.org level 2 cache > is 189k uncompressed. For comparison, four z9 raster tiles from > tile.osm.org are about 200kb, a fifth of the bytes transferred. Retina > tiles would decrease the size different. The real benefit of client side rendering is, when the user can choose what to render on which zoom level. It is no problem, if it is now to slow on mobile phones or laptops. In two or three years it will be much faster. Now you have city names in cyrillic font in serbia/bulgaria or names in chinese font in china like in normal osm map. Let the user choose, if he want to see cyrillic, chinese or latin font on whole world. If the user can choose to see camping sites on zoom level 8 or on z14. About ten years ago, i wrote a simple client side rendering of osm in OpenGL as a desktop program (no browser). Stefan Ziegler. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
I've been busy with On 2020-05-24 10:26 p.m., Joseph Eisenberg wrote: Thank you for making this, it looks like a lot of work! These client-side vector tiles at z8? (https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html) My laptop (2015 Macbook Pro, 16 GB RAM, 2.2 GHz Intel Core i7) appears to use some effort to show areas with a large amount of data, For example if I view England + Wales on z8 and then move over to the Netherlands and then to Germany, at z7 and z8, CPU usage goes up to >100% for a time, and there is a noticeable delay. Perhaps part of this is a delay in serving the tiles? I turned on Content-Encoding based compression which cuts the bytes transferred in half. This helps significantly. In general vector tiles have more large tiles, even if the average tile size is the same. The largest z8 tile in the area is 1MB transfer size while an average weighted by requests on the tile.osm.org level 2 cache is 189k uncompressed. For comparison, four z9 raster tiles from tile.osm.org are about 200kb, a fifth of the bytes transferred. Retina tiles would decrease the size different. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On Sun, May 24, 2020 at 19:37 Paul Norman via dev wrote: ... out. Mapbox GL styles are written in JSON and most of the tools > overwrite any formatting. This means there's no way to add comments to Have you investigated HJSON? See: https://hjson.github.io/ The Raku language (https://raku.org) has a module that handles it. Blessings, -Tom ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On Tuesday 26 May 2020, nd...@redhazel.co.uk wrote: > > Rasterization is a data generalization technique too. No it isn't. Generalization is the most frequently abused term in cartography. Calling ST_Simplify and way_area filtering on OSM landcover polygon data generalization is a bit like calling JPEG compression noise reduction. But we are drifting off topic here. > Yes, in some > cases it leaves no artifacts at all Who would say that? In fact sampling artefacts (a.k.a. aliasing) are the mother of all artefacts in computer graphics. The point is that sampling artefacts are very well understood and it is known how to minimize them. > I've been looking into the code and I couldn't find anything related > to polygon generalization. Where is it done? Lossy geometry compression is happening in ST_AsMVTGeom - which is not just the inevitable coordinate rounding to the vector tile coordinate grid but also the subjective choice of per feature Douglas-Peucker line compression. See https://github.com/postgis/postgis/pull/463 for the well argued suggestion not to do that. The way_area filtering is explicit in the SQL code. -- Christoph Hormann http://www.imagico.de/ ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On 26/05/2020 13:26, Christoph Hormann wrote: As Joseph has hinted when it comes to accurate rendering for precise mapper feedback without unintentional incentives at the lower zoom levels there are other options based on pre-rasterizing the data which would still allow using the benefits of client side styling. I have demonstrated that in the past. Rasterization is a data generalization technique too. Yes, in some cases it leaves no artifacts at all but that's not true in a general case and almost never true on mobile platforms. It is a viable solution but to make it work well the style should not use borders or bake in antialiasing (i.e. leave that to the client code). I am concerned about something else - even with polygon generalization in place, the performance is not great. Zooming/panning is rather slow on my not so weak PC - the map would be unusable on a mobile device or at best it would be draining the battery quickly. I've been looking into the code and I couldn't find anything related to polygon generalization. Where is it done? If it happens automatically in the Mapbox GL renderer that's too late - all the cost of preparing tiles, transmitting them, processing them in the client has already been incurred. Displaying data via WebGL is probably the cheapest part of the processing chain. ndrw ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On Tuesday 26 May 2020, Tom Hughes via dev wrote: > More specifically what you are seeing is that it will let you zoom in > beyond zoom 8 but you are seeing data that was simplified on the > basis that it would be shown at zoom 8 where those artefacts would > not be visible. As said that is not quite true - but of course attributing artefacts, noise and rendering distortions that are visible to specific data processing steps is not easy. As Joseph has hinted when it comes to accurate rendering for precise mapper feedback without unintentional incentives at the lower zoom levels there are other options based on pre-rasterizing the data which would still allow using the benefits of client side styling. I have demonstrated that in the past. -- Christoph Hormann http://www.imagico.de/ ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On 2020-05-26 10:59, Simon Poole wrote: I think you are missing "Only zoom 0 to 8 has been implemented so far. I started at zoom 0 and am working my way down." Oh zoom 8 is way further out than I was, my bad. The simplifications are not visible at that zoomlevel. Maarten Simon Am 26.05.2020 um 10:42 schrieb Maarten Deen: On 2020-05-26 10:15, Paul Norman via dev wrote: On 2020-05-25 1:15 a.m., Maarten Deen wrote: Still, it looks to be a very simplified subset of the complete data. So I'd be interested to see how this works on a full dataset. No, it has the full data except for admin boundaries and bay/straight names on zoom 7 and 8. When you compare it to OpenStreetMap Carto they're pretty similar. There are some labeling differences with sizes of labels and when labels come in, but those don't have anything to do with performance or size. No, that's not what I'm seeing. I am seeing very basic and rectangular landuses, obviously simplified, along with waterways, railways, motorways, trunk, primary secondary roads and they too have been simplified, to the extent that parallel ways on motorways are now crossing eachother. This is in my area: https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html#12.14/51.36196/6.08322 https://imgur.com/a/WcYlVUc Regards, Maarten ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
More specifically what you are seeing is that it will let you zoom in beyond zoom 8 but you are seeing data that was simplified on the basis that it would be shown at zoom 8 where those artefacts would not be visible. Tom On 26/05/2020 09:59, Simon Poole wrote: I think you are missing "Only zoom 0 to 8 has been implemented so far. I started at zoom 0 and am working my way down." Simon Am 26.05.2020 um 10:42 schrieb Maarten Deen: On 2020-05-26 10:15, Paul Norman via dev wrote: On 2020-05-25 1:15 a.m., Maarten Deen wrote: Still, it looks to be a very simplified subset of the complete data. So I'd be interested to see how this works on a full dataset. No, it has the full data except for admin boundaries and bay/straight names on zoom 7 and 8. When you compare it to OpenStreetMap Carto they're pretty similar. There are some labeling differences with sizes of labels and when labels come in, but those don't have anything to do with performance or size. No, that's not what I'm seeing. I am seeing very basic and rectangular landuses, obviously simplified, along with waterways, railways, motorways, trunk, primary secondary roads and they too have been simplified, to the extent that parallel ways on motorways are now crossing eachother. This is in my area: https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html#12.14/51.36196/6.08322 https://imgur.com/a/WcYlVUc Regards, Maarten ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
I think you are missing "Only zoom 0 to 8 has been implemented so far. I started at zoom 0 and am working my way down." Simon Am 26.05.2020 um 10:42 schrieb Maarten Deen: > On 2020-05-26 10:15, Paul Norman via dev wrote: >> On 2020-05-25 1:15 a.m., Maarten Deen wrote: >>> Still, it looks to be a very simplified subset of the complete data. >>> So I'd be interested to see how this works on a full dataset. >> >> No, it has the full data except for admin boundaries and bay/straight >> names on zoom 7 and 8. When you compare it to OpenStreetMap Carto >> they're pretty similar. There are some labeling differences with sizes >> of labels and when labels come in, but those don't have anything to do >> with performance or size. > > No, that's not what I'm seeing. I am seeing very basic and rectangular > landuses, obviously simplified, along with waterways, railways, > motorways, trunk, primary secondary roads and they too have been > simplified, to the extent that parallel ways on motorways are now > crossing eachother. > > This is in my area: > https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html#12.14/51.36196/6.08322 > https://imgur.com/a/WcYlVUc > > Regards, > Maarten > > ___ > dev mailing list > dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/dev signature.asc Description: OpenPGP digital signature ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On 2020-05-26 10:15, Paul Norman via dev wrote: On 2020-05-25 1:15 a.m., Maarten Deen wrote: Still, it looks to be a very simplified subset of the complete data. So I'd be interested to see how this works on a full dataset. No, it has the full data except for admin boundaries and bay/straight names on zoom 7 and 8. When you compare it to OpenStreetMap Carto they're pretty similar. There are some labeling differences with sizes of labels and when labels come in, but those don't have anything to do with performance or size. No, that's not what I'm seeing. I am seeing very basic and rectangular landuses, obviously simplified, along with waterways, railways, motorways, trunk, primary secondary roads and they too have been simplified, to the extent that parallel ways on motorways are now crossing eachother. This is in my area: https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html#12.14/51.36196/6.08322 https://imgur.com/a/WcYlVUc Regards, Maarten ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On 2020-05-25 1:15 a.m., Maarten Deen wrote: Still, it looks to be a very simplified subset of the complete data. So I'd be interested to see how this works on a full dataset. No, it has the full data except for admin boundaries and bay/straight names on zoom 7 and 8. When you compare it to OpenStreetMap Carto they're pretty similar. There are some labeling differences with sizes of labels and when labels come in, but those don't have anything to do with performance or size. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On 2020-05-26 06:24, Yves wrote: Le 26 mai 2020 03:58:50 GMT+02:00, Paul Norman via dev a écrit : Performance with normal basemaps and small stylesheets should be acceptable. Reading this, I don't think I understand. However it makes me think of a raster basemap plus a vector overlay for labels to solve internationalization issue. How lightweight such an overlay could be? You might run into problems when you have to place labels very exactly. For low zoom that is not an issue, if a place name is a bit left or right than that's no problem, but when you get to high zoom levels and want to place lets say street names, than you have to know where to place it and that means you have to know how the raster tile behind it is rendered. Regards, Maarten ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
It seems like the biggest problem with the vector rendering, for performance and cartography, is rendering areas such as water and landcover, while the big benefit is with text labels, SVG icons, and linear features. Is it possible to continue rendering the landcover and water areas from a pre-rendered raster file with this toolchain? – Joseph Eisenberg On Mon, May 25, 2020 at 9:27 PM Yves wrote: > > > Le 26 mai 2020 03:58:50 GMT+02:00, Paul Norman via dev < > dev@openstreetmap.org> a écrit : > > >Performance with normal basemaps and small stylesheets should be > >acceptable. > > Reading this, I don't think I understand. However it makes me think of a > raster basemap plus a vector overlay for labels to solve > internationalization issue. How lightweight such an overlay could be? > Yves > > ___ > dev mailing list > dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
Le 26 mai 2020 03:58:50 GMT+02:00, Paul Norman via dev a écrit : >Performance with normal basemaps and small stylesheets should be >acceptable. Reading this, I don't think I understand. However it makes me think of a raster basemap plus a vector overlay for labels to solve internationalization issue. How lightweight such an overlay could be? Yves ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On 2020-05-25 9:25 a.m., Jason Remillard wrote: Hi Paul, The Mapbox GL styles are very low level, and are like the mapnik XML input files. The Mapbox GL editing tools available don't seem to provide abstractions over the underlying specification. Yes, the specification and tools are quite clearly not designed for human readability. There has been some work at languages that compile to Mapbox GL styles, but nothing that supports expressions yet. Also, rewriting from scratch the main OpenStreetMap Carto style sheet is not appealing. There has been a lot of work put into it, and whatever happens next, everybody is going to be living with it for a long time. I'm a maintainer of OpenStreetMap Carto so I'm familiar with its development. I don't think this is a problem. It's not from scratch, it's implementing existing cartography in a new language. I have done some projects with Mapbox GL with client side rendering, and the performance on my tiny style sheet was unacceptable on mobile. Having the main style sheet use client side rendering will need to wait for newer and faster phones. Performance with normal basemaps and small stylesheets should be acceptable. The question is, how do you get normal sized tiles at low zooms from OSM data, how big will the tiles be at high zooms, and will the stylesheet be too complex. Going back to the problems. Long term it is clear that it would be a good thing if OSM had a client side rendering option. Also, right now, the existing tile servers are overloaded. A victim of the projects ongoing success and CartoCSS baked in performance issues. No, the issues have been with the tile CDN, not the tile servers. CartoCSS has never been a performance issue. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
Hi Paul, The Mapbox GL styles are very low level, and are like the mapnik XML input files. The Mapbox GL editing tools available don't seem to provide abstractions over the underlying specification. Also, rewriting from scratch the main OpenStreetMap Carto style sheet is not appealing. There has been a lot of work put into it, and whatever happens next, everybody is going to be living with it for a long time. I have done some projects with Mapbox GL with client side rendering, and the performance on my tiny style sheet was unacceptable on mobile. Having the main style sheet use client side rendering will need to wait for newer and faster phones. Going back to the problems. Long term it is clear that it would be a good thing if OSM had a client side rendering option. Also, right now, the existing tile servers are overloaded. A victim of the projects ongoing success and CartoCSS baked in performance issues. My ambitious thought is that the equivalent of the CartoCSS needs to be invented to provide a human friendly interface to both the Mapbox GL and mapnik rendering engines. The language should be at a higher level of abstraction. For example, it should just handle the "layer" key. To broaden the pool of contributors, it should directly use the OSM key and values, and not a foreign intermediate SQL database or tile schema. Lastly, accept that the language is just for rendering normal maps, and assume things like labels are always on top. I am not a fan of CartoCSS, but in theory, we could write a new CartoCSS processor to target Mapbox GL to save the main style sheet. Jason ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On 25/05/2020 01:34, Paul Norman via dev wrote: Landuse, vegetation, and other natural features are not rendered until zoom 7. This is the scale of OpenStreetMap Carto zoom 8, and these features first appear at zoom 5. There are numerous problems with unprocessed OpenStreetMap data at these scales. OpenStreetMap Carto gets a result that looks acceptable but is poor at conveying information by tweaking Mapnik image rasterizing options. I'm looking for better options here involving preprocessed data, but haven't found any. I think we will need fairly heavy pre-processing (filtering and simplification/generalization) of data before they hit the tile and the client. Carto is doing some of it already, even though data stay on the server side at all times. With the client-side rendering this is much more important for three reason - real time rendering, network traffic, limited resources on client side. Some ideas: - Tiles should contains only objects to be rendered, so all the filtering currently done in Carto should be applied at the tile construction time. - Currently Carto produces a lot of data that are later filtered out in Mapnik (via spatial occlusion) - this also should be brought back to a tiling script. - Tiles should be split into layers (similar to layers in Carto). Then the user would have an option to select layers they are interested in (think language versions, POIs). With more, smaller tiles there is also a potential for distributing the server side load more uniformly. - For z5-z12 we will need some kind of polygon generalization/simplification. For example, at z7 I can see all landuses in England, with some towns occupying only a couple of px but but still being comprised of hundreds of landuse polygons. I'm not talking about anything fancy, but merging anything sub-1px into "pixel rectangles" could go a long way. (Edit: I see you are already doing it - I can see some simplified landuse polygons when zooming in further). The technology choices are designed to be suitable for a replacement for tile.osm.org. This means minutely updates, high traffic, high reliability, and multiple servers. Do we need minutely updates for vector tiles? Sure, that's a nice feature to have but with client-side rendering any performance issues will be directly exposed to the users. And performance is a big issue in most client-side rendering solutions. If we can keep the current raster backend with minutely updates for mapping feedback, I have no problem with hourly or daily updates for the vector version. We would need a raster option for browsers/devices not supporting WebGL anyway. ndrw ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On Monday 25 May 2020, Paul Norman via dev wrote: > https://github.com/pnorman/openstreetmap-cartographic. In general this looks like a good approach to evaluate where the practical issues are without being bogged down by bloated legacy tools (well - except for the client side of course). > Landuse, vegetation, and other natural features are not rendered > until zoom 7. This is the scale of OpenStreetMap Carto zoom 8, and > these features first appear at zoom 5. There are numerous problems > with unprocessed OpenStreetMap data at these scales. OpenStreetMap > Carto gets a result that looks acceptable but is poor at conveying > information by tweaking Mapnik image rasterizing options. I'm looking > for better options here involving preprocessed data, but haven't > found any. Yes, i think this style as is is a good demonstration that the current approach used universally in the industry (and that is used in ST_AsMVT() as well) of performing per feature lossy vector geometry compression (usually in combination with way_area filtering) is a dead end. You can see that nicely if you overzoom the demo map: http://imagico.de/files/screenshot_cartographic.png http://imagico.de/files/screenshot_carto.png This creates a lot more artefacts and noise in the rendering than the AGG rendering issues in Mapnik typically do even at the resolution it is intended for - including counterproductively adding a lot of high frequency noise in many cases. > Often forgotten is the development requirements. The style needs to > support multiple developers working on similar areas, git merge > conflicts while maintaining an easy development workflow. I'm still > figuring this out. Mapbox GL styles are written in JSON and most of > the tools overwrite any formatting. This means there's no way to add > comments to lines of codes. Comments are a requirement for a style > like this, so I'm investigating minimal pre-processing options. The > downside to this will make it harder to use with existing GUI editors > like Fresco or Maputnik. I think this is a point that cannot be overstated. Mapbox GL JSON (or FWIW JSON in general) in contrast to CartoCSS is a software developer centered format and not a cartographer centered format. Many mainstream low sophistication users of client side vector tile frameworks focus exclusively on interactive editors and not working on the code level - which as you explained is not an option for a cooperative community project or for styles with a higher level of sophistication. Just re-casting the JSON data structures in a different format easier to edit by hand, suitable for comments and leading to well readable diffs is IMO not enough for a truly cartographer centered approach. But this is definitely a hard problem because there are so many in parts opposite requirements. Having a reference renderer that does not build on an extremely complex and platform dependent framework (a.k.a. web browser) would be important for style development - i think Joseph also hinted in that direction. It would also be important for things like automated tests etc. One other thing to keep in mind - OSM-Carto uses polygon fill patterns a lot for differentiating different types of polygons. While it is in principle possible to show fill patterns in continuous zooming in an ergonomic fashion i have so far never seen maps in the wild doing that. This would be something to consider if you want to create a style similar in the richness of landcover rendering to OSM-Carto. -- Christoph Hormann http://www.imagico.de/ ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On 2020-05-25 09:59, Paul Norman via dev wrote: On 2020-05-24 10:26 p.m., Joseph Eisenberg wrote: Thank you for making this, it looks like a lot of work! These client-side vector tiles at z8? (https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html) My laptop (2015 Macbook Pro, 16 GB RAM, 2.2 GHz Intel Core i7) appears to use some effort to show areas with a large amount of data, For example if I view England + Wales on z8 and then move over to the Netherlands and then to Germany, at z7 and z8, CPU usage goes up to >100% for a time, and there is a noticeable delay. Perhaps part of this is a delay in serving the tiles? Some of the tiles are decently large so could be slow to download, but based on your CPU it's not that. More likely it's the high number of vegetation and landuse features at those zooms. There are some things I might be able to do reduce the feature count. I'm not noticing much slowness on my computer, but having a lot of cores does help here I think. I won't notice it when my browser is using one core (out of six) at 100%. I did get some delay when zooming in, at first I thought "oh, only motorways, that's not much" but after some 10 seconds I got landuse and other roads. Still, it looks to be a very simplified subset of the complete data. So I'd be interested to see how this works on a full dataset. Regards, Maarten ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
On 2020-05-24 10:26 p.m., Joseph Eisenberg wrote: Thank you for making this, it looks like a lot of work! These client-side vector tiles at z8? (https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html) My laptop (2015 Macbook Pro, 16 GB RAM, 2.2 GHz Intel Core i7) appears to use some effort to show areas with a large amount of data, For example if I view England + Wales on z8 and then move over to the Netherlands and then to Germany, at z7 and z8, CPU usage goes up to >100% for a time, and there is a noticeable delay. Perhaps part of this is a delay in serving the tiles? Some of the tiles are decently large so could be slow to download, but based on your CPU it's not that. More likely it's the high number of vegetation and landuse features at those zooms. There are some things I might be able to do reduce the feature count. It would be nice to see a demonstration of this applied server-side to produce raster tiles. Are there any samples of that yet? There are a few examples of producing raster tiles from vector tiles and a Mapbox GL style. Mapbox does this with Mapbox GL Native on the server, but I don't believe they've released the exact software they use. https://github.com/maptiler/tileserver-gl/ is open-source software that generates raster tiles on demand. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenStreetMap Cartographic: A client-side rendered OpenStreetMap Carto
Thank you for making this, it looks like a lot of work! These client-side vector tiles at z8? ( https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html) My laptop (2015 Macbook Pro, 16 GB RAM, 2.2 GHz Intel Core i7) appears to use some effort to show areas with a large amount of data, For example if I view England + Wales on z8 and then move over to the Netherlands and then to Germany, at z7 and z8, CPU usage goes up to >100% for a time, and there is a noticeable delay. Perhaps part of this is a delay in serving the tiles? It would be nice to see a demonstration of this applied server-side to produce raster tiles. Are there any samples of that yet? – Joseph Eisenberg On Sun, May 24, 2020 at 5:37 PM Paul Norman via dev wrote: > I've been working on a new project, OpenStreetMap Cartographic. This is > a client-side rendering based on OpenStreetMap Carto. This is an > ambitious project, as OpenStreetMap Carto is an extremely complex style > which shows a large number of features. The technical choices I'm making > are designed so the style is capable of handling the load of osm.org > with minutely updates. > > I've put up a world-wide demo at > https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html, > using data from 2020-03-16, and you can view the code at > https://github.com/pnorman/openstreetmap-cartographic. > > Incomplete parts > > > Only zoom 0 to 8 has been implemented so far. I started at zoom 0 and am > working my way down. > > Admin boundaries are not implemented. OpenStreetMap Carto uses > Mapnik-specific tricks to deduplicate the rendering of these. I know how > I can do this, but it requires the changes I intend to make with the flex > backend. > > Landuse, vegetation, and other natural features are not rendered until > zoom 7. This is the scale of OpenStreetMap Carto zoom 8, and these > features first appear at zoom 5. There are numerous problems with > unprocessed OpenStreetMap data at these scales. OpenStreetMap Carto gets > a result that looks acceptable but is poor at conveying information by > tweaking Mapnik image rasterizing options. I'm looking for better options > here involving preprocessed data, but haven't found any. > > I'm still investigating how to best distribute sprites. > > Technology > == > > The technology choices are designed to be suitable for a replacement for > tile.osm.org. This means minutely updates, high traffic, high reliability, > and multiple servers. Tilekiln, the vector tile generator, supports all > of these. It's designed to better share the rendering results among > multiple servers, a significant flaw with renderd + mod_tile and the > standard filesystem storage. It uses PostGIS' ST_AsMVT, which is very > fast with PostGIS 3.0. On my home system it generates z0-z8 in under > 40 minutes. > > Often forgotten is the development requirements. The style needs to > support multiple developers working on similar areas, git merge conflicts > while maintaining an easy development workflow. I'm still figuring this > out. Mapbox GL styles are written in JSON and most of the tools > overwrite any formatting. This means there's no way to add comments to > lines of codes. Comments are a requirement for a style like this, so > I'm investigating minimal pre-processing options. The downside to this > will make it harder to use with existing GUI editors like Fresco or > Maputnik. > > Cartography > === > > The goal of this project isn't to do big cartography changes yet, but > client-side rendering opens up new tools. The biggest immediate change > is zoom is continuous, no longer an integer or fixed value. This means > parameters like sizes can smoothly change as you zoom in and out, > specified by their start and end size instead of having to specify each > zoom. > > Want to help? > = > > Have a look at https://github.com/pnorman/openstreetmap-cartographic and > have a go at setting it up and generating your own map. If you have > issues, open an issue or pull request. Or, because OpenStreetMap > Cartographic uses Tilekiln have a look at > https://github.com/pnorman/tilekiln/issues. > > > ___ > dev mailing list > dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
