Thanks for the replies Andrea and Jody. We’ve started using FME Server to load larger datasets directly into the database and then register them with GeoServer. We were just curious if there was some configuration we’d missed that would alleviate the issue when uploading directly via GeoServer with Postgres, but it sounds like there isn’t so we’ll stick with the FME process for now. Thanks again.
Kind regards, Peter Marlow From: Andrea Aime <andrea.a...@geosolutionsgroup.com> Sent: 20 December 2022 17:00 To: Marlow, Peter <peter.mar...@cgi.com> Cc: geoserver-users@lists.sourceforge.net Subject: Re: [Geoserver-users] Large file uploads to Postgres/PostGIS fail EXTERNAL SENDER: Do not click any links or open any attachments unless you trust the sender and know the content is safe. EXPÉDITEUR EXTERNE: Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe à moins qu’ils ne proviennent d’un expéditeur fiable, ou que vous ayez l'assurance que le contenu provient d'une source sûre. The GeoServer core configuration model is not fully thread safe. Whenever the REST API is used with a write method, it grabs a global lock that prevents concurrent usage of other REST API, as well as the Wicket UI, for the entire duration of the call. It protects the internal catalog, but works poorly if you are trying to upload larger datasets. There is a plugin, called "importer", that can do larger upload tasks without grabbing a lock... actually, I've never checked it, but that's a problem, in that it allows concurrent modification of the configuration, potentially causing issues... we should probably separate the data upload from the config changes, and grab the lock just around the config changes. That said, the same could be done on the REST API, assuming someone develops enough support for a REST method to advertise that it will handle its own locking. In both cases, some code changes are needed to best support your use case. Without them, go for the importer and hopefully you will not stumble into issues (that really depends on the rate at which these changes occur). Cheers Andrea On Tue, Dec 20, 2022 at 3:18 PM Marlow, Peter via Geoserver-users <geoserver-users@lists.sourceforge.net<mailto:geoserver-users@lists.sourceforge.net>> wrote: Hi, We are using GeoServer 2.18.2 and Postgres 11 with the PostGIS extension for our website. With small datasets we are not having any issues, but we are having trouble uploading larger datasets to GeoServer. After making a request, it hangs for a long time and ends up failing. When the request is made, the Geoserver web portal also becomes unresponsive and won’t allow menus to view layers, etc to be opened instead returning a server busy page. The file sizes that cause the issue tend to be ~15MB+ in size. Example request: curl -v -u [un]:[pw] -XPUT -H "Content-type: application/zip" --data-binary @dataset.zip http://localhost:8080/geoserver/rest/workspaces/geonode/datastores/[postgres_datastore]/file.shp<https://urldefense.com/v3/__http:/localhost:8080/geoserver/rest/workspaces/geonode/datastores/*5bpostgres_datastore*5d/file.shp__;JSU!!AaIhyw!oxVs7IqLqE1zV2CIMem39ggpYL0WZdN566Ge7Vn2QubQx4F3aomVSYBLl9wZPzxaAnpz-QyLtItvg6zL6MWYfopDCyiTkQ$> To narrow down the problem we tried connecting to a Postgres 9 database with our existing setup to see if the issue was particular to the version of Postgres. When that didn’t work we installed vanilla versions Postgres 11 and GeoServer 2.22.0 locally. This was to test if isolating GeoServer and Postgres from our existing deployment and using a later version of Geoserver would help, but the issue persists. We now believe that there may be something wrong with the way GeoServer is adding data to Postgres databases. Is there any configuration or setup we’re missing that might help alleviate this issue? Kind regards, Peter Marlow | Lead Agile Engineer, Central Government | CGI Methuen Park, Chippenham SN14 0GB peter.mar...@cgi.com<mailto:peter.mar...@cgi.com> | cgi-group.co.uk<https://urldefense.proofpoint.com/v2/url?u=https-3A__eur01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Furldefense.proofpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-5Feur01.safelinks.protection.outlook.com-5F-2D3Furl-2D3Dhttps-2D253A-2D252F-2D252Furldefense.proofpoint.com-2D252Fv2-2D252Furl-2D253Fu-2D253Dhttp-2D2D3A-2D5F-2D5Fcgi-2D2D2Dgroup.co.uk-2D2526d-2D253DDwQFAg-2D2526c-2D253DH50I6Bh8SW87d-2D5FbXfZP-2D5F8g-2D2526r-2D253DYqptHQRpueLxpzhhdrs9rNv0AFhjpvKOJXE5QFg5sec-2D2526m-2D253D-2D5Fgj4eu3TogA5s7Z5B1WbkD7clWbl5NNaUmVrxA-2D5F5Zeg-2D2526s-2D253DTz-2D2DxcS3aG5tkV6h1vHL3UawEi8RBzV8kOEw90UlDdpU-2D2526e-2D253D-2D26data-2D3D02-2D257C01-2D257CJenna.Harris004-2D2540gov.wales-2D257C877c495f30814889579908d803b34f61-2D257Ca2cc36c592804ae78887d06dab89216b-2D257C0-2D257C0-2D257C637263415952110161-2D26sdata-2D3DEe4sImOacbMBaz4O3Ab89Y-2D252BfaxW3GgL4H1GGBmyOa0c-2D253D-2D26reserved-2D3D0-2526d-253DDwMFAw-2526c-253DH50I6Bh8SW87d-5FbXfZP-5F8g-2526r-253DYqptHQRpueLxpzhhdrs9rNv0AFhjpvKOJXE5QFg5sec-2526m-253Ddrz7xRm6eCLkEFCVsBotApQMeCL8KWSA3S2h31qSqM8-2526s-253DJqe-2DNjizHGdWjDjQiSFHkgY-5FcK5Z5crpYPpVxS1VKnA-2526e-253D-26data-3D02-257C01-257CJenna.Harris004-2540gov.wales-257C83f63c4967c749281cbe08d803b3d312-257Ca2cc36c592804ae78887d06dab89216b-257C0-257C0-257C637263418159776608-26sdata-3DrXHcmDMVu2wQTcPw79lSlrOHqeTL1AeJY4-252BnrfEgw2I-253D-26reserved-3D0&d=DwMFAw&c=H50I6Bh8SW87d_bXfZP_8g&r=YqptHQRpueLxpzhhdrs9rNv0AFhjpvKOJXE5QFg5sec&m=LBD2yGv39o8eh4_mmKvmQA40U0MTVXFpTllp2GEizGk&s=sR8k_roT2qxNDgh72mYFIxySjFs4DU6JN3TqtlfQRlU&e=> _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/<https://urldefense.com/v3/__http:/www.ianturton.com/talks/foss4g.html*/__;Iw!!AaIhyw!oxVs7IqLqE1zV2CIMem39ggpYL0WZdN566Ge7Vn2QubQx4F3aomVSYBLl9wZPzxaAnpz-QyLtItvg6zL6MWYforsVZYMRw$> - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html<https://urldefense.com/v3/__http:/geoserver.org/comm/userlist-guidelines.html__;!!AaIhyw!oxVs7IqLqE1zV2CIMem39ggpYL0WZdN566Ge7Vn2QubQx4F3aomVSYBLl9wZPzxaAnpz-QyLtItvg6zL6MWYfoo9FlJfqA$> If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer<https://urldefense.com/v3/__https:/github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer__;!!AaIhyw!oxVs7IqLqE1zV2CIMem39ggpYL0WZdN566Ge7Vn2QubQx4F3aomVSYBLl9wZPzxaAnpz-QyLtItvg6zL6MWYfopvn85-UA$> Geoserver-users@lists.sourceforge.net<mailto:Geoserver-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/geoserver-users<https://urldefense.com/v3/__https:/lists.sourceforge.net/lists/listinfo/geoserver-users__;!!AaIhyw!oxVs7IqLqE1zV2CIMem39ggpYL0WZdN566Ge7Vn2QubQx4F3aomVSYBLl9wZPzxaAnpz-QyLtItvg6zL6MWYfoqv5tzwIg$> -- Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://bit.ly/gs-services-us<https://urldefense.com/v3/__http:/bit.ly/gs-services-us__;!!AaIhyw!oxVs7IqLqE1zV2CIMem39ggpYL0WZdN566Ge7Vn2QubQx4F3aomVSYBLl9wZPzxaAnpz-QyLtItvg6zL6MWYfoq1E7_ymQ$> for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions Group phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 https://www.geosolutionsgroup.com/<https://urldefense.com/v3/__https:/www.geosolutionsgroup.com/__;!!AaIhyw!oxVs7IqLqE1zV2CIMem39ggpYL0WZdN566Ge7Vn2QubQx4F3aomVSYBLl9wZPzxaAnpz-QyLtItvg6zL6MWYfoq3eoXeBw$> http://twitter.com/geosolutions_it<https://urldefense.com/v3/__http:/twitter.com/geosolutions_it__;!!AaIhyw!oxVs7IqLqE1zV2CIMem39ggpYL0WZdN566Ge7Vn2QubQx4F3aomVSYBLl9wZPzxaAnpz-QyLtItvg6zL6MWYfooZARPA8Q$> ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail
_______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
