On Fri, Jan 8, 2021 at 4:42 AM Russell Grew < russell.g...@douglaspartners.com.au> wrote:
> All, > > > > I quite enjoyed the two OGC Testbed 16 videos prepared by GeoSolutions > Group. > > > > https://www.youtube.com/watch?v=hc81tlFMDbA > Thanks! > > > I have a few large (5+ GB) geopackage files hosted read only by GeoServer. > They could be shapefiles but I went with geopackage to do something > progressive, and avoid the 2 GB limit which some software has. > In our experience GeoHash sorting helped greatly on the 200GB GeoPackage, and only in a smaller amount on the 10GB ZoomStack package, where the generalized table extension had a far greater effect*. Your case might vary, depends a lot on the number of records (the more records, the more the effect is visible) and the type of disks (less visible on SSDs unless the size of the package is huge, more visible on spinning disks... I'm guessing network mounted disks also benefit). > Following Andre’s talk I wonder if I should attempt to reorder the data > in these files by geohash. To help it load faster. > > > > Is my understanding correct? Does anyone know of any tools to enable this? > I had to build the machinery from scratch in Java. Thinking out loud, if you want to use something else, maybe you can dump the data in postgresql, add the geohash column, and create a view that returns data sorted by geohash by default. Then use ogr2ogr to dump a geopackage from the view, that hopefully retains the right record order. > Perhaps I could use GeoServer 2.19 RC hosting my current files and then > dump them with gs:GeoPackage? If this is the way forward any advice on how > to tell it what layer I want to dump would be great. The WPS I have looked > at before all had vector layer process input boxes. > I've attached a request sorting by GeoHash against the ZoomStack data set, hopefully you can use it as a reference to build your request. If you try it out, let us know how the results perform. Cheers Andrea *: As a note, GeoServer has been extended to produce the packages with the new extensions, but has not ability to read them (the read part was in a different Testbed component, developed by another company). That said, the existing work on the write side would help develop the read portion quicker. == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- *Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.*
<?xml version="1.0" encoding="UTF-8"?> <wps:Execute xmlns:wps="http://www.opengis.net/wps/1.0.0"; xmlns="http://www.opengis.net/wps/1.0.0"; xmlns:gml="http://www.opengis.net/gml"; xmlns:ogc="http://www.opengis.net/ogc"; xmlns:ows="http://www.opengis.net/ows/1.1"; xmlns:wcs="http://www.opengis.net/wcs/1.1.1"; xmlns:wfs="http://www.opengis.net/wfs"; xmlns:xlink="http://www.w3.org/1999/xlink"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; version="1.0.0" service="WPS" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd";> <ows:Identifier>gs:GeoPackage</ows:Identifier> <wps:DataInputs> <wps:Input> <ows:Identifier>contents</ows:Identifier> <wps:Data> <wps:ComplexData mimeType="text/xml; subtype=geoserver/geopackage"> <![CDATA[ <geopackage xmlns="http://www.opengis.net/gpkg"; name="zoomstack-styles-sort-geohash"> <features name="land" identifier="land"> <description>land</description> <srs>EPSG:27700</srs> <featuretype>oszoom:land</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="national_parks" identifier="national_parks"> <description>national_parks</description> <srs>EPSG:27700</srs> <featuretype>oszoom:national_parks</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="urban_areas" identifier="urban_areas"> <description>urban_areas</description> <srs>EPSG:27700</srs> <featuretype>oszoom:urban_areas</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="sites" identifier="sites"> <description>sites</description> <srs>EPSG:27700</srs> <featuretype>oszoom:sites</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="greenspace" identifier="greenspace"> <description>greenspace</description> <srs>EPSG:27700</srs> <featuretype>oszoom:greenspace</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="woodland" identifier="woodland"> <description>woodland</description> <srs>EPSG:27700</srs> <featuretype>oszoom:woodland</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="contours" identifier="contours"> <description>contours</description> <srs>EPSG:27700</srs> <featuretype>oszoom:contours</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="district_buildings" identifier="district_buildings"> <description>district_buildings</description> <srs>EPSG:27700</srs> <featuretype>oszoom:district_buildings</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="local_buildings" identifier="local_buildings"> <description>local_buildings</description> <srs>EPSG:27700</srs> <featuretype>oszoom:local_buildings</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="surfacewater" identifier="surfacewater"> <description>surfacewater</description> <srs>EPSG:27700</srs> <featuretype>oszoom:surfacewater</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="foreshore" identifier="foreshore"> <description>foreshore</description> <srs>EPSG:27700</srs> <featuretype>oszoom:foreshore</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="waterlines" identifier="waterlines"> <description>waterlines</description> <srs>EPSG:27700</srs> <featuretype>oszoom:waterlines</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="roads_local" identifier="roads_local"> <description>roads_local</description> <srs>EPSG:27700</srs> <featuretype>oszoom:roads_local</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="roads_regional" identifier="roads_regional"> <description>roads_regional</description> <srs>EPSG:27700</srs> <featuretype>oszoom:roads_regional</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="roads_national" identifier="roads_national"> <description>roads_national</description> <srs>EPSG:27700</srs> <featuretype>oszoom:roads_national</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="rail" identifier="rail"> <description>rail</description> <srs>EPSG:27700</srs> <featuretype>oszoom:rail</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="boundaries" identifier="boundaries"> <description>boundaries</description> <srs>EPSG:27700</srs> <featuretype>oszoom:boundaries</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="etl" identifier="etl"> <description>etl</description> <srs>EPSG:27700</srs> <featuretype>oszoom:etl</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="railway_stations" identifier="railway_stations"> <description>railway_stations</description> <srs>EPSG:27700</srs> <featuretype>oszoom:railway_stations</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="airports" identifier="airports"> <description>airports</description> <srs>EPSG:27700</srs> <featuretype>oszoom:airports</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> <features name="names" identifier="names"> <description>names</description> <srs>EPSG:27700</srs> <featuretype>oszoom:names</featuretype> <sort xmlns:fes="http://www.opengis.net/fes/2.0";> <fes:SortProperty> <fes:ValueReference>geom</fes:ValueReference> </fes:SortProperty> </sort> <indexed>true</indexed> <styles>true</styles> </features> </geopackage> ]]> </wps:ComplexData> </wps:Data> </wps:Input> </wps:DataInputs> <wps:ResponseForm> <wps:RawDataOutput> <ows:Identifier>geopackage</ows:Identifier> </wps:RawDataOutput> </wps:ResponseForm> </wps:Execute>
_______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
