On Tue, Jul 5, 2011 at 10:54 PM, Yun-Hui Fan <[email protected]> wrote:
> Hi, all.
>
> My company has been experimenting with GeoServer (2.0.2 and 2.1) for
> several months. We use Ruby's Net::HTTP to publish image mosaics to the
> GeoServer via the REST API. Both the publishing code and the GeoServer are
> running on 64-bit RedHat 5 Linux (CentOS 5) with JDK 6u26. We have a cron
> that executes the publishing script once every minute, and we publish at
> least 10-20 mosaics per minute 24/7.
>
Wow, that's quite the workload. At 10 per minute it's 14400 new layers a
day, you would reach 5 million layers in a single year.
I've seen installations that have up to 160000 layers, but never one that
has millions. Not sure our current
catalog management is ready to deal with that amount of layers, we currently
store all layers configuration
in memory (that might not be a huge problem) but various searches do linear
scans over all the layers
(now this would be a problem when you have millions of layers).
I am wondering if your use case would not be better served with fewer
mosaics having time/elevation
support enabled:
http://geoserver.org/display/GEOS/GSIP+60+-+WMS+Time+and+Elevation+support+for+vector+and+raster+data
The work is currently being done on trunk.
>
> We have had resource problems since we started running GeoServer. The
> problems get worse when we publish more mosaics to the GeoServer. We
> finally narrowed it down to file descriptors/TCP sockets. We are also
> seeing memory growth (leak?) and process growth (growth of the number of
> processes or threads) that closely follow the growing trend of the open TCP
> sockets. We use Ganglia to monitor these metrics. To temporarily alleviate
> the problem, we raised the file descriptor limit from the default 1024 to
> somewhere in the 5-figure range. But then the memory limit was reached
> before the file descriptor limit was reached. We have tried running the
> GeoServer in either the jetty container that it's packaged with or a tomcat
> 6 container. We got the same resource problems.
>
> Other than monitoring the metrics graphically using Ganglia, we also use
> "netstat -anp --inet" to check the TCP sockets occasionally. Here is a
> snippet of the output:
>
> Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> tcp 0 0 0.0.0.0:53088 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:33920 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:38912 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:49408 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:44800 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:48288 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:39040 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:59776 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:39936 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:45632 0.0.0.0:*
> LISTEN 4404/java
> tcp 0 0 0.0.0.0:58816 0.0.0.0:*
> LISTEN 4404/java
>
> The number of them increases with the time we publish image mosaics to the
> GeoServer. We have had to restart the GeoServer once every few hours to
> clear out those sockets (and the memory).
>
This one is new to me. Are the mosaic stores over a NFS or other network
file system?
What I know is that 2.1.1 leaks file descriptors at every new mosaic
configuration or new
geotiff configuration, that is being fixed:
http://jira.codehaus.org/browse/GEOS-4656
(the fixes are being checked on trunk, we should backport them on 2.1.x
before 2.1.2 is released)
The issue has no known workarounds (you can try to force GC on the virtual
machine, that might
result in the files being closed, but it's just taking a chance and hoping
the GC involves the
input streams kept open).
Cheers
Andrea
--
-------------------------------------------------------
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 962313
http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf
-------------------------------------------------------
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Geoserver-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
