Hi Andrea,
Thanks for the detailed feedback.
So, I've decided to eliminate Oracle from the equation. I'm now accessing
OS Strategi from the shapefiles (locally sourced). Using the exact same
stylesheets as the Oracle ones.
With 1 thread from the shapefile using the PNG Encoder, the times are more
reasonable, with an average of 800ms per request (still pretty slow though).
But with 10 threads, the requests take anywhere from 5 to 14 times as long
to respond as they did with 1 thread - an average across the board of
7543ms per request. So the PNG Encoder doesn't appear to be scaling nicely.
CPU usage peaks at 20%.
===
I then re-did the Oracle tests (still using the PNG Encoder), just for this
layer (so everything is identical except the data source).
Average times were 2000ms with 1 thread and 8355ms for 10 threads.
------------
I then did all four tests again but with the original encoder. In 3 of the
four cases, on average the PNG Encoder was faster, but with 10 cores from
shapefile the original encoder was faster (7259ms for PNG, 7543ms for
default).
Conclusions:
The data source isn't the sole factor involved in the speed degredation.
PNG encoder doesn't seem to be scaling up well. I'm wondering if rather
than it being the data (it may be), it could also be the styles?
You can see all 8 results here: http://maps.warwickshire.gov.uk/misc/
As OS Strategi is a free dataset, I can provide the shapefiles and styles
and layergroup definitions; if you copy them into one of your instances you
can probably be up and testing inside of a few minutes.
Oh, and no SQL views here (we have a grand total of two of those, but
they're not here).
Regards,
Jonathan
On 23 December 2013 20:08, Andrea Aime <[email protected]> wrote:
> On Mon, Dec 23, 2013 at 6:55 PM, Jonathan Moules <
> [email protected]> wrote:
>
>> Hi Andrea,
>> I've put the images of the results, plus the csv files from jmeter here
>> if you want to check: http://maps.warwickshire.gov.uk/misc/
>>
>> That layer group comprises 6 other layer groups, each of which is
>> basically a different Ordnance Survey product set.
>> The super zoomed out stuff (1:150,000 outward) is all OS Strategi which
>> contains 15 layers - all Oracle. There's also a "EU basemap" group under it
>> which contains 3 layers (those are all local shapefiles).
>>
>
>> ====
>>
>> The Oracle datasource is certainly slow, but it seemed odd that it was
>> slower with the PNG encoder. - Also, as the requests are all for the same
>> area, wouldn't GeoServer/Oracle have cached the results (speeding things
>> up)?
>>
>
> Oracle should cache, as should the file system cache for the shapefiles.
> GeoServer only caches db connections, but never data
> (at least so far).
> As said, I find it hard to believe the PNG encoder has anything to do with
> the discrepancies you've found, the image
> encoding time for an image the size of a screen is way less than a second,
> your response times, even with the
> single thread, are like 3 seconds instead.
>
>
>
>> Swapping datasource:
>> If I test JMeter against the EU Basemap (3 local shapefiles) - PNG
>> Encoder, even using 50 threads, the CPU usage only goes up to 90%, with a
>> throughput of 63.8/s. The average response rate there is 747ms, compared to
>> one thread where it's just 136ms.
>>
>
> This is more like it. The response times of the Oracle based layers are
> not slow, they are a tragedy.
>
>
>> So I don't seem to be able to get 100% CPU use.
>>
>
> It's normal, you're on Windows, so you cannot use OpenJDK. The Oracle JDK
> rendering subsystem has internal synchronizations
> that prevent it to scale up nicely, that's why we suggest to install a
> GeoServer every 2-4 cores and load balance them instead.
>
>
>>
>> For the EU basemap, the sweet spot is around 15 threads (85% CPU use) -
>> 61.5/s; any more and geoserver starts lagging.
>>
>
> Yeah, see above about load balancing the GeoServer to get a higher
> throughput.
>
>
>>
>> Weird that I'm throwing about 4 times as many cores at the problem than
>> you are (based on your ~4 core VM), but still max out at 60 requests a
>> second. Co-incidence?
>>
>
> Most likely, my result is done with OpenJDK, with Oracle one I got around
> 40-50 instead, and the topp:states map, which has much less data, goes up
> to 95 r/s
>
>
>>
>> Sorry if I'm failing to understand something. I don't think I'm missing
>> anything obvious.
>>
>
> The issue is definitely with the Oracle access, but I don't know where...
> I mean, if you could
> serve at 60r/s all day continously (I know, not realistic, but let's have
> a ballpark) you'd be
> serving over 5 million requests a day.
> However, when you can serve only 8 requests a minute well... you could as
> well ship out paper maps instead.
> Now, Oracle is indeed a constant pain, but something this slow is truly
> problematic.
> We recently discovered an issue with sql views over Oracle, but afaik
> you're publishing straight tables
> instead, no?
>
> Cheers
> Andrea
>
> --
> *== GeoSolutions will be closed for seasonal holidays from 23/12/2013 to
> 06/01/2014 ==*
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via Poggio alle Viti 1187
> 55054 Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob: +39 339 8844549
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>
--
This transmission is intended for the named addressee(s) only and may
contain sensitive or protectively marked material up to RESTRICTED and
should be handled accordingly. Unless you are the named addressee (or
authorised to receive it for the addressee) you may not copy or use it, or
disclose it to anyone else. If you have received this transmission in error
please notify the sender immediately. All email traffic sent to or from us,
including without limitation all GCSX traffic, may be subject to recording
and/or monitoring in accordance with relevant legislation.
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Geoserver-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
