Just to announce that the search improvements have (finally) been submitted
to Sourceforge (trunk revision 1625).
The new code has been tested on Redhat 4u4 (icc 9.1) and Windows XP (VC++
Express 2008).
-----Original Message-----
From: Morrell Richard (external) [mailto:richard.morr...@uk.thalesgroup.com]
Sent: 20 January 2010 14:17
To: openslp-devel@lists.sourceforge.net
Subject: [Openslp-devel] Performance improvements for DAs with large numbers
of services
We have been having performance problems with our service-oriented system,
and I have been allocated a task to speed up searches, as some of our DAs
have over 500 services registered, and the linear scan was consuming too
much CPU. I have nearly completed the work, and should be ready to commit
it to the svn trunk over the next week or so. The changes to the database
and predicate modules are fairly extensive, but the new functionality is
controllable through the configuration file. This email is a "heads-up" for
the changes:
New module (slpd_index.c) to maintain an index that uses a balanced binary
tree for string values (boolean/integer/opaque/keyword are not indexed).
The index is used to reduce the number of entries to be tested, and hence
the CPU resources used and time taken. To do this, it stores additional
information, so there is an impact on the memory usage. The indexing can be
controlled through the configuration file as described below. A restriction
is imposed in that indexing properties cannot be changed by reloading the
configuration using SIGHUP, only by a full restart of the daemon.
Create and maintain an index on the service type, and use that index when
processing a search using a concrete service type, and when doing an
attribute retrieval, but only when enabled in the configuration file
(net.slp.indexSrvtype=true). We have concrete types that have few
duplicates, and we always perform an attribute retrieval for each service
returned, so this works well for us. Currently, the service type index is
used in preference to an attribute index so, if you have only a few discrete
concrete types, but an attribute with a large number of discrete values, it
may be better not to enable the service type index, or to use only abstract
types in searches.
Create and maintain indexes on attributes named in the configuration file
(net.slp.indexedAttributes=<attribute-name-1>,<attribute-name-2>,...), and
use these where appropriate. This means where there is a test for equality,
eg. '(location=Main Building Second Floor)' or '(location=Main Building*)',
for one of the named attributes (including wildcard searches as indicated)
as the only test in the predicate or as a test in a top-level '&' operation.
Parse the attribute string on registration, and store the result in the
database entry (avoids re-parsing the attribute string for every search, and
makes it easier to extract the data to update the indexes). This also
increases memory usage, and cannot currently be disabled.
Refactor the predicate code to create a parse tree for the predicate, and
use that when testing an entry for whether it matches (avoids the re-parsing
of the predicate string every time, and makes it easier to analyse the
predicate for whether to use an attribute index). This change, together
with the pre-parsing of attributes, speeds up searches by a factor of
several times even in the absence of indexes.
Our testing measured response time for an indexed search in an idle system
(assumed to be directly related to CPU load), and indicates that the
response is pretty constant (should be logarithmic in theory), regardless of
the number of services in the DA (tested up to 1000) - the main determinant
of response time is the number of matching entries found in the index (the
match set) that then have to be filtered by scope, predicate, security etc.
as before, so it works best with attributes that have large numbers of
discrete values (so only a small number are matched), or where most of a
match set is required (so only a small number of entries are tested and
rejected). Nearly all our searches fall into one of these two categories.
Richard Morrell
_____
Software Architecture & Technologies
Tel:
+ 44 (0)161 741 3426
Email:
richard.morr...@uk.thalesgroup.com
<mailto:richard.morr...@uk.thalesgroup.com>
Intranet:
Website <http://uwsweb.tus.fr.thales/functions/operations/SAT/index.stm> :
Products <http://satweb.tus.uk.thales/products.php> : Helpdesk
<http://sat-helpdesk.tus.fr.thales/>
THALES UNDERWATER SYSTEMS LTD
Dolphin House, Ashurst Drive
Cheadle Heath, Stockport SK3 OXB - UK
<http://www.thales-naval.com/> www.thales-naval.com
Thales Underwater Systems Limited. Registered in England
and Wales. Registered number 3084140.
Registered office 2 Dashwood Lang Road, The Bourne Business Park,
Addlestone, Nr. Weybridge, Surrey KT15 2NX
This email, including any attachment, is a confidential communication
intended solely for the use of the individual or entity to whom it is
addressed. It contains information which is private and may be proprietary
or covered by legal professional privilege. If you have received this email
in error, please notify the sender upon receipt, and immediately delete it
from your system.
Anything contained in this email that is not connected with the businesses
of this company is neither endorsed by nor is the liability of this company.
Whilst we have taken reasonable precautions to ensure that any attachment to
this email has been swept for viruses, we cannot accept liability for any
damage sustained as a result of software viruses, and would advise that you
carry out your own virus checks before opening any attachment.
This email, including any attachment, is a confidential communication
intended solely for the use of the individual or entity to whom it is
addressed. It contains information which is private and may be proprietary
or covered by legal professional privilege. If you have received this email
in error, please notify the sender upon receipt, and immediately delete it
from your system.
Anything contained in this email that is not connected with the businesses
of this company is neither endorsed by nor is the liability of this company.
Whilst we have taken reasonable precautions to ensure that any attachment to
this email has been swept for viruses, we cannot accept liability for any
damage sustained as a result of software viruses, and would advise that you
carry out your own virus checks before opening any attachment.
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Openslp-devel mailing list
Openslp-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/openslp-devel
