Re: [Geoserver-users] slow vector tiles
The draw primitives don’t have topology. They’re just path/fill. As a trivial example, if you get a vector tile with a line up the middle, would your client code style the left or right side as water? Brad From: Matt Hakim Sent: Saturday, 17 November 2018 8:16 PM To: andrea.a...@geo-solutions.it Cc: geoserver-users@lists.sourceforge.net Subject: Re: [Geoserver-users] slow vector tiles Many thanks for your reply. I followed your thread on the devel list as well. We are using Vector tiles so we can apply styling on the client (openlayers) when a feature is selected. For our scenario, if the output Vector tile just included the Polygons that would have been rendered into the PNG anyway then that would be fine (I think). I had a bit of a look through the geotools StreamingRenderer.java and it seems to deal with simplifying geometries and rendering shape primitives using java's Graphics2D, StreamingRenderer interestingly also calls TopologyPreservingSimplifier (similar to Vector output tiles). Conceptually, could you capture the draw primitive requests going to Graphics2D and instead output them in Vector tile format? Wouldn't that work at the same speed as the Raster output tiles (or even faster as no styling is applied)? Obviously I'm dreaming here (and it would take me months to understand StreamingRenderer...) In the meantime we simplified our geometry to 10% of the original size which while slow, maintains the health of our server for the initial tile generation after which they are cached anyway. Matt. On Sun, Nov 11, 2018 at 8:41 PM Andrea Aime mailto:andrea.a...@geo-solutions.it> > wrote: Hi, I've downloaded the dataset and a first glance at it confirmed that it taking a long time to display is, I'm afraid, just normal, at least with a dataset having a single geometry with over 1.5 million points. You have a very detailed australian coastline there, and vector tiles require the geometry in output, inside the vector tile, to be topologically correct. Doing a topology preserving geometry simplification and clipping against a source geometry having this many points is just going to take a lot of time. Rendering in WMS instead we can use much more efficient algorithms that do not preserve the geometry topology, hence the "instant" response time. I think I've spotted a bit that could be made faster, but I am not the maintainer of the vector tiles modules, so going to discuss it on the devel list, but generally speaking, I'd try to avoid having to deal with such complex polygons in vector tiles caching, and if you do, just be patient and let the machinery take the time needed to build the tiles. A quick trick to improve things could be to split this large polygons into smaller ones, for example by pre-clipping it over a regular grid (OpenStreeMap has been doing this for years to make world coastlines manageable). Cheers Andrea On Thu, Nov 8, 2018 at 8:16 AM Matt Hakim mailto:thep...@gmail.com> > wrote: We are having an issue with very slow (>60s) generation of Vector tiles (mapbox) via open layers TMS calls. This can be reproduced through the geoserver web interface on the "Tile Layers" page. The same tiles seem to be generated very quickly in the PNG format. We are running geoserver 2.14 on linux with 2GB heap. The vector tile generation appears to create a lot of garbage - so much that sometimes "OutOfMemory: heap" or "OutOfMemory: gc time exceeded" messages appear in the log. Steps to reproduce: 1. Install geoserver 2.14 and Vector tile output extension 2. Download shapefile "Australia (AUS) ASGS Ed 2016 Digital Boundaries in ESRI Shapefile Format" from web site: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1270.0.55.001July%202016?OpenDocument 3. Load above shapefile into a datastore & enable "mapbox" tile output format. 4. Navigate to the "Tile Layers" web page in geoserver. For the layer created from the above shapefile, choose "mapbox" in the Preview dropdown. 5. Preview will display albeit extremely slowly. 6. You may need to restart geoserver. 7. Perform step 4 above instead choosing "png" in Preview dropdown. 8. Preview displays instantly. Any ideas? ___ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.
Re: [Geoserver-users] slow vector tiles
On Sat, Nov 17, 2018 at 10:16 AM Matt Hakim wrote: > Many thanks for your reply. I followed your thread on the devel list as > well. > > We are using Vector tiles so we can apply styling on the client > (openlayers) when a feature is selected. > For our scenario, if the output Vector tile just included the Polygons > that would have been rendered into the PNG anyway then that would be fine > (I think). > I had a bit of a look through the geotools StreamingRenderer.java and it > seems to deal with simplifying geometries and rendering shape primitives > using java's Graphics2D, StreamingRenderer interestingly also calls > TopologyPreservingSimplifier (similar to Vector output tiles). > Normal simplification is not topology preserving, see the Decimator class, it brutal, efficient and the geometries that come out of it are barely usable for rendering (but definitely not topologically valid) The only case you can find using the TopologyPreservingSimplifier is for building polygon offsets, in that case we need to start with a topologically valid geometry to compute the buffer. > Conceptually, could you capture the draw primitive requests going to > Graphics2D and instead output them in Vector tile format? Wouldn't that > work at the same speed as the Raster output tiles (or even faster as no > styling is applied)? Obviously I'm dreaming here (and it would take me > months to understand StreamingRenderer...) > I did not write the vector tiles module, had nothing to do with it actually, but if you check, it's using a lot of the rendering chain functionality. There is quite a bit of chatter about vector tiles efficiency on the net these weeks, you'll find that others are doing a non topological preserving simplification followed by a "make valid" operation, which tries to fix the errors. JTS, the library we use for geometries, does not provide such operation, but if you want you could look in the GEOS MakeValid operator (used by postgis for example) and replicate enough of it to use it in MVT, or sponsor someone (http://geoserver.org/support/) to do it on your behalf, well, that would be very welcomed. In parallel, I'm discussing on the devel list about doing clipping before simplification, in my tests it speeds up your use case quite a bit (mind, still slow). Cheers Andrea == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- *Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.* ___ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
Re: [Geoserver-users] slow vector tiles
Many thanks for your reply. I followed your thread on the devel list as well. We are using Vector tiles so we can apply styling on the client (openlayers) when a feature is selected. For our scenario, if the output Vector tile just included the Polygons that would have been rendered into the PNG anyway then that would be fine (I think). I had a bit of a look through the geotools StreamingRenderer.java and it seems to deal with simplifying geometries and rendering shape primitives using java's Graphics2D, StreamingRenderer interestingly also calls TopologyPreservingSimplifier (similar to Vector output tiles). Conceptually, could you capture the draw primitive requests going to Graphics2D and instead output them in Vector tile format? Wouldn't that work at the same speed as the Raster output tiles (or even faster as no styling is applied)? Obviously I'm dreaming here (and it would take me months to understand StreamingRenderer...) In the meantime we simplified our geometry to 10% of the original size which while slow, maintains the health of our server for the initial tile generation after which they are cached anyway. Matt. On Sun, Nov 11, 2018 at 8:41 PM Andrea Aime wrote: > Hi, > I've downloaded the dataset and a first glance at it confirmed that it > taking a long time to display is, I'm afraid, > just normal, at least with a dataset having a single geometry with over > 1.5 million points. > > You have a very detailed australian coastline there, and vector tiles > require the geometry in output, inside the vector > tile, to be topologically correct. Doing a topology preserving geometry > simplification and clipping against a source geometry having this many > points is just going to take a lot of time. > > Rendering in WMS instead we can use much more efficient algorithms that do > not preserve the geometry topology, hence the "instant" response time. > > I think I've spotted a bit that could be made faster, but I am not the > maintainer of the vector tiles modules, so going to discuss it on the > devel list, but generally speaking, I'd try to avoid having to deal with > such complex polygons in vector tiles caching, and if you do, > just be patient and let the machinery take the time needed to build the > tiles. > > A quick trick to improve things could be to split this large polygons into > smaller ones, for example by pre-clipping it over a regular grid > (OpenStreeMap has been doing this for years to make world coastlines > manageable). > > Cheers > Andrea > > > > On Thu, Nov 8, 2018 at 8:16 AM Matt Hakim wrote: > >> >> We are having an issue with very slow (>60s) generation of Vector tiles >> (mapbox) via open layers TMS calls. This can be reproduced through the >> geoserver web interface on the "Tile Layers" page. The same tiles seem to >> be generated very quickly in the PNG format. >> >> We are running geoserver 2.14 on linux with 2GB heap. The vector tile >> generation appears to create a lot of garbage - so much that sometimes >> "OutOfMemory: heap" or "OutOfMemory: gc time exceeded" messages appear in >> the log. >> >> Steps to reproduce: >> 1. Install geoserver 2.14 and Vector tile output extension >> 2. Download shapefile "Australia (AUS) ASGS Ed 2016 Digital Boundaries >> in ESRI Shapefile Format" from web site: >> http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1270.0.55.001July%202016?OpenDocument >> 3. Load above shapefile into a datastore & enable "mapbox" tile output >> format. >> 4. Navigate to the "Tile Layers" web page in geoserver. For the layer >> created from the above shapefile, choose "mapbox" in the Preview dropdown. >> 5. Preview will display albeit extremely slowly. >> 6. You may need to restart geoserver. >> 7. Perform step 4 above instead choosing "png" in Preview dropdown. >> 8. Preview displays instantly. >> >> Any ideas? >> >> >> >> ___ >> Geoserver-users mailing list >> >> Please make sure you read the following two resources before posting to >> this list: >> - Earning your support instead of buying it, but Ian Turton: >> http://www.ianturton.com/talks/foss4g.html#/ >> - The GeoServer user list posting guidelines: >> http://geoserver.org/comm/userlist-guidelines.html >> >> If you want to request a feature or an improvement, also see this: >> https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer >> >> >> Geoserver-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/geoserver-users >> > > > -- > > Regards, Andrea Aime == GeoServer Professional Services from the experts! > Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime > @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 > Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 > 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it > ---
