Search performance testing
A series of test were run to determine how quickly searches for
businesses and business services were executed given the same number of
services in a node while varying the number of business under which the
services were published.
This was done by repeatedly filling an empty node with the following
numbers of businesses and services/business:
# of Businesses
# of Services/Business
100,000
1
10,000
10
1,000
100
100
1000
We see that the total number of services is consistently 100,000 for
every instance of the node. Each business and service was published
under a lone unique names so that every search only has one result. Load
test repeating the TFindServiceBM scenario was done to measure mean
response times. The TFindServiceBM scenario searches the node by name
for a random service. The name is supplied by a previously generated
file containing the names of all services. The mean times for each ~
30min load test is illustrated in the table below (the tests were run
twice on each node instance):
Arrival rate \ businesses 100,000 10,000 1,000 100
100,000 247ms
<http://xlt.xceptance.de/download/results/juddi/20090821-130234/> 250ms
<http://xlt.xceptance.de/download/results/juddi/20090819-183924/> 327ms
<http://xlt.xceptance.de/download/results/juddi/20090821-093748/> 231ms
<http://xlt.xceptance.de/download/results/juddi/20090821-164739/>
100,000 186ms
<http://xlt.xceptance.de/download/results/juddi/20090825-131253/> 242ms
<http://xlt.xceptance.de/download/results/juddi/20090824-181227/> 243ms
<http://xlt.xceptance.de/download/results/juddi/20090821-093748/> 238ms
<http://xlt.xceptance.de/download/results/juddi/20090824-102539/>
Clicking on any of the mean times above will link to the XLT report of
run. We can see that the resulting mean times are almost always ~ 245ms
except for a couple of outliers. This would suggest that searches for
services are independent of how they are distributed over businesses.
As for find_business operations, a 100,000 transaction/hour constant
arrival rate could not be maintained for all the different distributions
of services. Instead an arrival rate of 6,000 t/h was used since this
seemed to be close to highest load the slowest configuration (100
businesses) could take. The table below shows the mean times for the
TFindBusinessBM runs:
Arrival rate \ businesses 100,000 10,000 1,000 100
6,000 64ms
<http://xlt.xceptance.de/download/results/juddi/20090825-122527/> 27ms
<http://xlt.xceptance.de/download/results/juddi/20090824-173745/> 91ms
<http://xlt.xceptance.de/download/results/juddi/20090824-160655/> 869ms
<http://xlt.xceptance.de/download/results/juddi/20090824-121054/>
TFindBusinessBM finds and returns a random business much in the same way
as TFindServiceBM, by searching for a business name supplied by a
previously generated file containing the names of all businesses in the
node. Here we see an increase in mean response time which seams to be
related to the increase in services /business. This is probably cause by
the lengthening returned SOAP message that includes not only details
off the business but also a list of all the services associated with it.
The first value of the node with 100,000 businesses with just one
service each, does not seem to support this trend but that may be
because of the relatively large number of businesses that must be
searched through. More testing would be needed to determine this.
Here are trend reports showing performance differences between service
searches
<http://xlt.xceptance.de/download/results/juddi/trendreport-20090826-131138/requests.html#request-summary>
and businesses searches
<http://xlt.xceptance.de/download/results/juddi/trendreport-20090826-130737/>.
System setup:
This
<http://people.apache.org/repo/m2-snapshot-repository/org/apache/juddi/juddi-portal-bundle/3.0.0.SNAPSHOT/juddi-portal-bundle-3.0.0.20090723.201427-7.zip>
release of jUDDI ran on the following:
Processor: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz
RAM: 2011 MB
OS: Ubuntu 9.04, linux kernel 2.6.28-13-generic
This
<http://xlt.xceptance.de/download/results/juddi/testsuites/20090821-juddiTestSuite.zip>
test suite was used to run the above test.
Please let me know if there are any specific scenarios you, the jUDDI
developers, would like to see.
Jeremi
