Steve,
Sorry for the confusion. This was not in response to you, but Andrea
about dropping the module from geoserver altogether. I understand your
decision because I have also found that jdbcconfig is a problematic
bottle neck (but as I mentioned, mostly because of the poor cache
support and the unnecessarily repetitive queries).
In the attachment you can find the code for exporting jdbc to hard
drive. The file needs to be temporarily added to the class path and
scanned by the spring appcontext, then when you are logged in as an
admin you can call /jdbcexport from the browser to force the caching of
the catalog. I hope you can use this somehow.
Kind Regards
Niels
On 15-11-17 12:51, Steve Omondi wrote:
Oh really, drop it? I thought jdbcconfig was the recommended way
for clustering and is particularly useful when you are working
with huge catalogs. I assumed it was actually being used in
production by people.
Our intention is not to drop it entirely. But to run more or less a
catalog that is based on XML for some specific workspaces that we
really need to be performant urgently.
While we have found jdbcconfig to be an excellent solution for
clustering, it
also
has made *3 Load balanced Geoserver 2.11.2 to be 6 times slower than 1
Geoserver 2.9* all other configurations kept constant.
And this is an issue that we can't ignore in PROD for our case.
We intent to move a few of the workspaces and layers especially
complex LayerGoups fron the JDBC catalog to achieve better performance
over convenience of the DB Catalog.
*Otherwise, JDBCConfig is an excellent module in use*.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Kind regards,
Steve Omondi
On Wed, Nov 15, 2017 at 1:30 PM, Niels Charlier <ni...@scitus.be
<mailto:ni...@scitus.be>> wrote:
Oh really, drop it? I thought jdbcconfig was the recommended way
for clustering and is particularly useful when you are working
with huge catalogs. I assumed it was actually being used in
production by people.
The main problem seems to be the performance issues though, as
this email also proves, as well as both of the two PR's that are
open who are related to the same thing. But I believe this is
rather solvable and would make a huge difference.
I can review these PRs no problem, since they are actually related
to the work I am doing for this module.
Perhaps these improvements there can blow some new life into it.
Regards
Niels
On 14-11-17 20:10, Andrea Aime wrote:
Hi Niels,
today during the PSC meeting the topic came out of whether we
should just drop
JDBCConfig, since:
* mails related to it are not really getting answered
* pull requests related to it are sitting there not getting
reviewed
We still have to see if there is some interested, but we need
someone to champion the module
just enough to make it worth keeping around (I'm not talking
about immediate answers, it's
a community/unsupported module, but at least some presence).
I'm also going to ping the people making pull requests.
Cheers
Andrea
On Tue, Nov 14, 2017 at 5:24 PM, Niels Charlier <ni...@scitus.be
<mailto:ni...@scitus.be>> wrote:
Hello Steve,
First I'd like to say I have been doing some work on
considerable performance improvements for jdbcconfig. It
happens to be the case that jdbcconfig doesn't take good use
of its cache and repeatedly sends the same queries over and
over again. It looks promising but I still need to do some
improvements and write some tests and I had other work coming
on top of it. But this is definitely coming.
Also, I wrote some code in geoserver for myself to
conveniently export the jdbc catalog back to the file system.
I found a way to do this easily by firing a bunch of catalog
change events to the file system catalog. However, I am not
sure to make it easily available to end users at this point,
I assume that just giving you the code won't be very helpful
for you. Perhaps we can work something out if you are interested.
Regards
Niels
On 09-11-17 11:02, Steve Omondi wrote:
Hi guys,
I have *JDBCConfig* enabled and connected to a Postgres DB.
I have over 500 layers spread across different workspaces
and layergroups.
After running performance tests for GetMap requests against
this configuration versus XML Catalog based geoserver I
really need to revert my catalog back to XML in GEOSERVER
DAT DIR and probably share it using NFS.
*Is their a method to export the Database Catalog (based on
JDBCConfig) back to XML Catalog?*
Kind regards,
Steve Omondi
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list
Please make sure you read the following two resources before posting to
this list:
- Earning your support instead of buying it, but Ian
Turton:http://www.ianturton.com/talks/foss4g.html#/
<http://www.ianturton.com/talks/foss4g.html#/>
- The GeoServer user list posting
guidelines:http://geoserver.org/comm/userlist-guidelines.html
<http://geoserver.org/comm/userlist-guidelines.html>
Geoserver-users@lists.sourceforge.net
<mailto:Geoserver-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geoserver-users
<https://lists.sourceforge.net/lists/listinfo/geoserver-users>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's
most engaging tech sites, Slashdot.org!
http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list Please make sure you read the
following two resources before posting to this list: -
Earning your support instead of buying it, but Ian Turton:
http://www.ianturton.com/talks/foss4g.html#/
<http://www.ianturton.com/talks/foss4g.html#/> - The
GeoServer user list posting guidelines:
http://geoserver.org/comm/userlist-guidelines.html
<http://geoserver.org/comm/userlist-guidelines.html>
Geoserver-users@lists.sourceforge.net
<mailto:Geoserver-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geoserver-users
<https://lists.sourceforge.net/lists/listinfo/geoserver-users>
--
Regards,
Andrea Aime
==GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.==Ing. Andrea Aime
<https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa&entry=gmail&source=g>@geowolfTechnical
LeadGeoSolutions S.A.S.Via di Montramito 3/A
<https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa&entry=gmail&source=g>55054
Massarosa
<https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa&entry=gmail&source=g>
(LU)phone: +39 0584 962313fax: +39 0584 1660272mob: +39 339
8844549http://www.geo-solutions.ithttp://twitter.com/geosolutions_it
<http://twitter.com/geosolutions_it>
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta
elettronica e/o nel/i file/s allegato/i sono da considerarsi
strettamente riservate. Il loro utilizzo è consentito
esclusivamente al destinatario del messaggio, per le finalità
indicate nel messaggio stesso. Qualora riceviate questo messaggio
senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del
messaggio stesso, cancellandolo dal Vostro sistema. Conservare il
messaggio stesso, divulgarlo anche in parte, distribuirlo ad
altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.
The information in this message and/or attachments, is intended
solely for the attention and use of the named addressee(s) and
may be confidential or proprietary in nature or covered by the
provisions of privacy act (Legislative Decree June, 30 2003,
no.196 - Italy's New Data Protection Code).Any use not in accord
with its purpose, any disclosure, reproduction, copying,
distribution, or either dissemination, either whole or partial,
is strictly forbidden except previous formal approval of the
named addressee(s). If you are not the intended recipient, please
contact immediately the sender by telephone, fax or e-mail and
delete the information in this message that has been received in
error. The sender does not give any warranty or accept liability
as the content, accuracy or completeness of sent messages and
accepts no responsibility for changes made after they were sent
or for other risks which arise as a result of e-mail
transmission, viruses, etc.
package org.geoserver.jdbcstore.rest;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import java.util.logging.Logger;
import org.geoserver.catalog.Catalog;
import org.geoserver.catalog.CatalogInfo;
import org.geoserver.catalog.LayerGroupInfo;
import org.geoserver.catalog.LayerInfo;
import org.geoserver.catalog.MapInfo;
import org.geoserver.catalog.NamespaceInfo;
import org.geoserver.catalog.ResourceInfo;
import org.geoserver.catalog.StoreInfo;
import org.geoserver.catalog.StyleInfo;
import org.geoserver.catalog.WorkspaceInfo;
import org.geoserver.catalog.event.impl.CatalogAddEventImpl;
import org.geoserver.catalog.event.impl.CatalogModifyEventImpl;
import org.geoserver.catalog.util.CloseableIterator;
import org.geoserver.config.GeoServer;
import org.geoserver.config.GeoServerPersister;
import org.geoserver.config.ServiceInfo;
import org.geoserver.config.ServicePersister;
import org.geoserver.config.util.XStreamPersisterFactory;
import org.geoserver.config.util.XStreamServiceLoader;
import org.geoserver.platform.GeoServerExtensions;
import org.geoserver.platform.GeoServerResourceLoader;
import org.geoserver.platform.resource.Resource;
import org.geoserver.platform.resource.Resource.Type;
import org.geoserver.security.impl.GeoServerRole;
import org.geotools.util.logging.Logging;
import org.opengis.filter.Filter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.security.access.AccessDeniedException;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
@Controller
@RequestMapping("/jdbcexport")
public class JdbcExport {
static Logger LOGGER = Logging.getLogger( "org.geoserver" );
@Autowired
GeoServer gs;
@Autowired
Catalog catalog;
@Autowired
GeoServerResourceLoader loader;
private <T extends CatalogInfo> void exportType(Class<T> clazz, GeoServerPersister gp) {
CloseableIterator<T> it = catalog.list(clazz, Filter.INCLUDE);
while (it.hasNext()) {
CatalogAddEventImpl event = new CatalogAddEventImpl();
event.setSource(it.next());
gp.handleAddEvent(event);
}
}
void cache(Resource res) throws IOException {
if (res.getType() == Type.DIRECTORY) {
res.dir();
for (Resource child : res.list()) {
cache(child);
};
} else {
res.file();
}
}
@GetMapping
public String execute() throws IOException {
if (!SecurityContextHolder.getContext().getAuthentication().getAuthorities()
.contains(GeoServerRole.ADMIN_ROLE)) {
throw new AccessDeniedException("You must be administrator.");
}
//export jdbc config to jdbc store
GeoServerPersister gp = new GeoServerPersister(loader,
new XStreamPersisterFactory().createXMLPersister());
//download jdbc catalog
exportType(NamespaceInfo.class, gp);
exportType(WorkspaceInfo.class, gp);
exportType(StoreInfo.class, gp);
exportType(ResourceInfo.class, gp);
exportType(LayerInfo.class, gp);
exportType(LayerGroupInfo.class, gp);
exportType(LayerInfo.class, gp);
exportType(LayerGroupInfo.class, gp);
exportType(StyleInfo.class, gp);
exportType(MapInfo.class, gp);
gp.handlePostGlobalChange(gs.getGlobal());
gp.handlePostLoggingChange(gs.getLogging());
gp.handleSettingsPostModified(gs.getSettings());
CatalogModifyEventImpl event = new CatalogModifyEventImpl();
event.setSource(catalog);
event.setPropertyNames(Collections.singletonList("defaultWorkspace"));
event.setNewValues(Collections.singletonList(catalog.getDefaultWorkspace()));
gp.handleModifyEvent(event);
@SuppressWarnings("rawtypes")
final List<XStreamServiceLoader> loaders =
GeoServerExtensions.extensions(XStreamServiceLoader.class);
ServicePersister sp = new ServicePersister(loaders, gs);
for (ServiceInfo si : gs.getServices()) {
sp.handlePostServiceChange(si);
}
return "/";
}
}
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list
Please make sure you read the following two resources before posting to this
list:
- Earning your support instead of buying it, but Ian Turton:
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines:
http://geoserver.org/comm/userlist-guidelines.html
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
