I'm load testing a 2.8 geoserver. Geoserver runs in tomcat on the backend of
an apache web server processing requests thru AJP. I have a jmeter load test
setup to send WMS GetMap requests (to get a JPEG image) for 40 users
continually. The test was running for about 10 seconds and processing hundreds
of requests and then failed with an Out of Memory exception. I doubled the
heapsize from 1024MB to 2048MB and it ran for longer and then still reported
Out of Memory.
But the problem would appear to be threads and not memory...my exception reads:
<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE
ServiceExceptionReport SYSTEM
"https://svcdev1.byers.com:443/geoserver/schemas/wms/1.1.1/WMS_exception_1_1_1.dtd";>
<ServiceExceptionReport version="1.1.1" > <ServiceException>
java.lang.OutOfMemoryError: unable to create new native thread
unable to create new native thread
</ServiceException></ServiceExceptionReport>
The code is apparently running out of threads, not memory. This is related to
the timer threads that WMS creates to detect the rendering taking too long.
My tomcat thread pool is sized to 50 threads and I've confirmed that only that
many are created. But by doing a threaddump while the system is running under
this jmeter load I see many more threads getting created which are simply the
timer threads each request is creating. For example I might see 50 threads
that are blocked on the Oracle database while there are 157 of the timer
threads.
It would appear that the timers get canceled but the timer thread itself does
not find an opportunity to exit/die before more timer threads are started.
Creating a Timer for every GetMap request would seem to be an inefficient way
to attack the problem which invites this issue - there is apparently no
Thread.join in scope to ensure these threads die.
How can I get a more graceful failure under load? In general my policy with
load is to size the tomcat thread pool so as to not have too many threads for
reasonable throughput. Then if more requests are received apache will say the
server is overloaded/retry. I don't want various other exceptions that are
supposed to indicate too much load but require judgement to that effect by
non-technical users.
Walter Stovall - Byers Engineering Company
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list
Please make sure you read the following two resources before posting to this
list:
- Earning your support instead of buying it, but Ian Turton:
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines:
http://geoserver.org/comm/userlist-guidelines.html
[email protected]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
