Re: [Geoserver-users] GeoServer Slow to Start with Lots of Namespaces( > 3h )

[Andrea Aime]

Hi,
I've tried to setup locally an artificial test case with lots of layers,
checking if by any chance it's just that
which is making things "bad".
Local setup:

   - GeoServer development series (2.11.x)
   - Postgresql/Postgis
   - One store, one database
   - 1 layers, each one with its own style


- Startup time: 21:45:20
- All layers, store and styles loaded at 21:45:43
- Startup complete at 21:45:59
> total startup time, less than 30 seconds

Now, by this I'm inclined to assume the issue is with having not a lot of
layers, but a lot of stores,
and the sanity check that GeoServer does when loading the store (there are
none for the layers per se).
The thing is, there is nothing new about these checks, they have been there
for 10+ years.. it
may be that what we do during this check has become more expensive recently.

But it also points out clearly that a random setup with lots of layers does
not incurs in the slow
startup, and there is a need for more details on how to reproduce.

Please everybody involved, share:

   - Version of GeoServer that starts fast, version that starts slow
   - How many workspaces
   - How many stores per workspace, and which type. Is there any type that
   feels slower than others?
   - If the store has the schema configured, or left blank
   - When configuring a new layer out of the store, how many tables are
   showing up?


Cheers
Andrea


On Fri, Dec 2, 2016 at 2:53 AM,  wrote:

> Hi Andrea,
>
>
>
> I filed a ticket since I think this symptom is very similar to the one I
> posted a couple of days ago. I talked with Andy who experienced similar
> issue but found no solution. Hope this get fixed soon.
>
>
>
> Warm regards,
>
> 신상희 드림
>
> [1] https://osgeo-org.atlassian.net/browse/GEOS-7884
>
> ---
> Shin, Sanghee
> Gaia3D, Inc. - The GeoSpatial Company
> http://www.gaia3d.com
>
>
>
> *보낸 사람: *Andrea Aime 
> *보낸 날짜: *2016년 12월 2일 금요일 오전 9:49
> *받는 사람: *Jaroslav Urik 
> *참조: *GeoServer Mailing List List 
> *제목: *Re: [Geoserver-users] GeoServer Slow to Start with Lots of
> Namespaces( > 3h )
>
>
>
> Hi,
>
> we don't know why it's happening but you're not the first to report it.
>
> I already asked in vain for someone to report a ticket with some details
> at https://osgeo-org.atlassian.net/projects/GEOS/issues ?
>
> If there is no bug report, for sure nobody will look into it when the
> monthly bug fix code sprint comes (if there is it's still no guarantee,
>
> but at least a chance).
>
>
>
> Cheers
>
> Andrea
>
>
>
> On Thu, Dec 1, 2016 at 2:36 PM, Jaroslav Urik 
> wrote:
>
> Hi,
>
> I have the same problem as described in [1] -- I am running GeoServer
> under Tomcat with a lot of georeferenced maps ( about 65 000 ) and it takes
> very long time to start - about 3-4 hours.
>
> The problem seems to be in that GeoServer is checking all of those
> resources one by one each time it starts..
>
> My question is: Is there any way to disable these checks? Or do them only
> one a month or so?
>
> Or is there any other solution?
>
> It is quite annoying when server needs to be restarted and people are
> waiting for it..
>
>
>
> Thanks in advance
>
>
>
> Jaroslav Urik
>
>
>
>
>
> [1] http://osgeo-org.1560.x6.nabble.com/GeoServer-Slow-to-
> Start-with-Lots-of-Namespaces-td5068101.html
>
>
> 
> --
>
> ___
> Geoserver-users mailing list
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>
>
>
>
> --
>
> ==
>
> GeoServer Professional Services from the experts! Visit
>
> http://goo.gl/it488V for more information.
>
> ==
>
>
>
> Ing. Andrea Aime
>
> @geowolf
>
> Technical Lead
>
>
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
>
> phone: +39 0584 962313 <+39%200584%20962313>
>
> fax: +39 0584 1660272 <+39%200584%20166%200272>
>
> mob: +39  339 8844549 <+39%20339%20884%204549>
>
>
>
> http://www.geo-solutions.it
>
> http://twitter.com/geosolutions_it
>
>
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is 

Re: [Geoserver-users] GetFeatureInfo with info_format=application/vnd.ogc.gml/3.1.1 causes memory leak

[Andrea Aime]

On Fri, Dec 2, 2016 at 11:06 AM, Jan De Moerloose <
jan.demoerlo...@geosparc.com> wrote:

> Hi,
>
> i think i may have found a memory leak in the WMS service of geoserver. It
> occurs when executing multiple (identical) GetFeatureInfo requests on one
> of the demo layers.
> The way to reproduce it is as follows:
>
> - deploy the latest stable geoserver war (2.10.0) on tomcat
> - execute the following curl command
>
> curl "127.0.0.1:8080/geoserver/ows?SERVICE=WMS=1.1.1&
> REQUEST=GetFeatureInfo=image%2Fpng=true&
> QUERY_LAYERS=tiger%3Apoly_landmarks=tiger%
> 3Apoly_landmarks_FORMAT=application%2Fvnd.ogc.gml%2F3.
> 1.1_COUNT=50=50=50=EPSG%3A4326=101&
> HEIGHT=101=-73.97317886352539%2C40.81266403198242%2C-73.
> 93850326538086%2C40.84733963012695
> =[1-3000]" > out.txt
>
> With 1G of memory, the server comes to a halt after about 1000 requests.
> A heap dump shows that memory is occupied by eclipse schema classes (emf).
>
> A search on the issue tracker did not reveal anything. Any ideas or could
> anyone try to reproduce ?
>

Our GML encoder is unfortunately schema driven, meaning we need to build a
in memory xml schema for it to work.

I'm not 100% sure, but I'm afraid the developer originally building the GML
output format for feature into created the leak by
building over and over a WFS schema in memory, that unfortunately causes
the leak
(due to how XSD and EMF work, each WFS schema instance gets referred back
from the GML schemas, which are singletons).

If my hypothesis is correct, the code should be amended to refer to
the wfsXsd-1.1 and xmlConfiguration-1.1 stored in the
spring context instead or re-creating them over and over, acting in fashion
similar to the WFS GML3OutputFormat class.

WMS does already depend on the WFS module in that very class by imports, so
adding a wiring at the spring level is not
going to cause more issues.

Is that something you can try to build?
In any case, I'd start by opening a ticket with steps to reproduce at
https://osgeo-org.atlassian.net/projects/GEOS/summary

Cheers
Andrea


-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---
--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Re: [Geoserver-users] Memory Leak seemingly caused by Monitoring plugin

[Andrea Aime]

On Fri, Dec 2, 2016 at 7:11 PM, Jason Newmoyer  wrote:

> Setup is: RHEL 6, GeoServer (war) 2.10.0, Jetty 9.3, PostgreSQL 9.4,
> monitoring plugin with hibernate enabled, configured for JNDI datasource
> provided by Jetty c3p0
>

I know that saving with hibernate tends to become slower and slower and ...
slower, but that is related to the inserts themselves
on the table, checking for the primary key uniqueness becomes progressively
slower until it becomes completely
unusable. That's why, at least in my company, we never use the hibernate
storage, but enable audit mode to dump
on file instead, and then transfer the info into elasticsearch using
logstash.

Database storage progressive slowdown should be fixable by removing the
primary key, however that is probably going to require
other changes, I have vague memories that the code might saving a record
and then updating it later, which might not work
without a primary key to refer to for the update statement.


>
> System is under pretty heavy use (1000's of requests per hour) and while
> the extension does what its intended to do, we end up with an unstable
> GeoServer after several hours and must be restarted. System reports
> resident memory slowly increasing unbounded for the java process. At about
> 2x the max heap size setting it starts to fall over. Services become
> unreachable and log starts printing huge lists of threads. Even the
> shutdown signal to Jetty seems to be ignored and the process must be
> SIGKILL'd.
>

You should collect a memory dump using jmap when things start getting bad
and see what's filling the memory.
Or just get one using Yourkit, they do have a 15 days trials license which
is normally good enough for a one-off analysis.


> I'm curious if this has been reported before if anyone can provide leads
> as to the cause. I checked JIRA and didn't see anything. I'm willing to
> peak around in the monitor code.
>
> Also, a lot of our users and using KML superoverlays and this causes about
> 90% of our requests to be operation='dispatch' which seem completely
> useless to me. Anyone know what the purpose of that is? I'm interested in
> making a configuration parameter to ignore these.
>

I'm not sure about this one... You probably want to check somewhere around
here:
https://github.com/geoserver/geoserver/blob/master/src/extension/monitor/core/src/main/java/org/geoserver/monitor/ows/MonitorCallback.java#L77
although you'll probably also end up looking into the main dispatcher here:
https://github.com/geoserver/geoserver/blob/master/src/ows/src/main/java/org/geoserver/ows/Dispatcher.java

I'm making a guess here, maybe you are hitting some GWC service? Those are
not playing by the same rules as the GeoServer inner ones
and I believe that could be a reason why monitoring has troubles figuring
out the details

Cheers
Andrea

-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---

Re: [Geoserver-users] Geoserver 2.10 handling of Oracle Multipoint Geometry features (Geoserver BUG?)

[Andrea Aime]

Hi,
looks like we have a patch but nobody bothered to write a test. Without one
the patch won't go in.
My guess is that it's just hard to find someone that has access to Oracle
and is also willing to complete
the work in their spare time.

Cheers
Andrea

On Fri, Dec 2, 2016 at 11:14 PM, Chris M  wrote:

> We are using Geoserver to serve Oracle Multipoint geometries as GeoJSON
> for use in Leaflet maps. The problem is for a multipoint oracle geometry
> only one (the first) point is returned in the GeoJSON.
>
> I have not been able to find much regarding this specific issue by
> searching the Geoserver Mailing List Archives, however, this has been
> mentioned in the GeoTools Issue Tracker here:
>
> https://osgeo-org.atlassian.net/browse/GEOT-5018
>
> Here is an example of our sdo_geometry as stored in Oracle:
>
> MDSYS.SDO_GEOMETRY(2005,4326,NULL,MDSYS.SDO_ELEM_INFO_
> ARRAY(1,1,1,3,1,1,5,1,1,7,1,1,9,1,1,11,1,1,13,1,1,15,1,1,17,
> 1,1,19,1,1,21,1,1,23,1,1,25,1,1,27,1,1,29,1,1),MDSYS.SDO_
> ORDINATE_ARRAY(-94.5460377524086,20.6982790126949,-90.4738630431407,31.
> 2499984169595,-90.455139056471,31.2500183780065,-90.4768843004608,31.
> 2475569741882,-90.4677293254458,31.2475623172449,-90.
> 4635038599556,31.247567709336,-90.4741218614148,31.2474940749101,-90.
> 4594171576172,31.2465043475159,-90.4523872566151,31.245139626753,
> -90.453889962981,31.2449028444609,-90.45355210797,31.2436239106601,-90.
> 4532291894738,31.2423062682822,-90.4462047630709,31.2321101141883,-90.
> 4529960800564,31.2298086565983,-90.4633401444988,31.2263420787051))
>
>  And here is the GeoJSON as generated by GeoServer:
>
> {"type":"FeatureCollection","totalFeatures":1,"features":[{
> "type":"Feature","id":"FMS_TEST.32778","geometry":{"type"
> :"MultiPoint","coordinates":[[-94.5460377524086,20.
> 6982790126949]]},"geometry_name":"GEOMETRY","properties":
> {"PROJECT_NO":"100829","PROJECT_DET":1000,"EXT_COR_
> PROJ_NO":"STP-0310-00(007)","CONV_PROJ_NO":"48-0310-00-007-
> 10","PLAN_LINK":"4333_STP-0310(7)A","DELIVERY":"05","URL":"http://
> some_link.com","SMB":"some_directory","P2":"http://anotherlink.com
> ","
> PDPM":null,"CREATE_OPER_ID":"RWD","CREATE_DATE":"2010-09-
> 29T19:40:34Z","UPDATE_OPER_ID":"RWD","UPDATE_DATE":"2013-04-
> 16T17:32:02Z","ENV":null,"DEPT":1}}],"crs":{"type":"
> EPSG","properties":{"code":"4326"}}}
>
>
> I am wondering if this should be submitted (as a bug?) on the Geoserver
> side as well.
>
> Any update/advice is greatly appreciated,
>
> Chris
>
> see also:
>
> https://osgeo-org.atlassian.net/browse/GEOT-5018
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Geoserver-users mailing list
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>


-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were