Ontotext are pleased to announce the release of OWLIM version 5.0
<http://www.ontotext.com/owlim> featuring a new transaction mechanism,
performance improvements, SPARQL 1.1 graph store protocol, integration
with TopBraid Composer/Live
<http://www.topquadrant.com/products/TB_Suite.html> and many other
improvements. The single most important new feature is the new
transaction management mechanism which allows for much *more reliable
and efficient handling of workloads where queries from multiple clients
are combined with frequent updates* of the data. As benchmark results
<http://www.ontotext.com/owlim/benchmark-results/owlim-5> demonstrate,
OWLIM 5.0 is *43% faster* than v.4.3 on the BSBM Explore and Update
<http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/>
scenario. As a result of several changes in the index structures, OWLIM
now requires *between 25% and 70% less storage space*.
Some of the most important improvements are listed below:
* *Transaction management and isolation mechanisms* have been
completely refactored. The previous strategy used lazy writing of
modified database pages, such that dirty pages were only flushed to
disk when further updates occur and no more memory is available.
While extremely fast, the problem with this approach is that there
is a considerable recovery time associated with replaying the
transaction log after an abnormal termination. The new mechanism
uses two modes: 'bulk-loading' (fast) with similar behaviour to
previous versions and 'normal' (safe) where database modifications
are flushed to disk as part of the commit operation. When running in
safe mode, *database recovery is instant* and there is a
*significant improvement in concurrency between updates and queries*.
* *New context indices* can be used to improve query performance when
data is modelled using many named graphs. These are switched on and
off using a single configuration parameter enable-context-index
* The *SPARQL 1.1 Graph Store HTTP Protocol* is now supported
according to the W3C Working Draft
<http://www.w3.org/TR/sparql11-http-rdf-update/> from the 12th May
2011. This provides a REST interface for managing collections of
graphs, using either directly or indirectly named graphs.
* *Sesame <http://www.openrdf.org>* *2.6.5* with many bug-fixes and
updates to bring SPARQL 1.1 Query
<http://www.w3.org/TR/2012/WD-sparql11-query-20120105/> support up
to the latest W3C Working Draft from the 5th January 2012.
* *Significant reduction in disk-space requirements* is achieved with
the following modifications:
o *Index compression* can now be used to reduce disk storage
requirements by using zip compression on database pages. This
feature if off by default, but can be switched on when creating
a new repository. The configuration parameter
index-compression-ratio can be set to -1 (the default value
indicating no compression) or a value in the range 10-50
<https://confluence.ontotext.com/pages/createpage.action?spaceKey=OWLIMint&title=10-50&linkCreation=true&fromPageId=17596523>
indicating the desired percentage reduction in page sizes. Any
pages that can not be compressed by the specified amount are
stored uncompressed. Therefore a compression ratio that is too
aggressive will not bring many benefits. Experiments have shown
that for large datasets a value of about 30% is close to optimal
and leads to a total disk space saving of around 50%.
o *Restructuring of the triple indices* has also led to a
reduction in disk-space requirements of around 18% independent
of the compression functionality
o *Entity compression* is a modification that reduces the storage
requirements for the lookup table that maps between internal
identifiers and resources. This is transparent to the user and
happens automatically. More disk space reductions are apparent
using this version.
* A new *literal index* is created automatically for numeric and
date/time data-types. The index is used during query evaluation if a
query or a sub-query (e.g. union) has a filter that is comprised of
a conjunction of literal constraints, e.g. FILTER(?x >= 3 && ?y <= 5
&& ?start > "2001-01-01"^^xsd:date). Other patterns, including those
that use negation, will not use the index for this version of OWLIM.
* Tighter integration with TopQuadrant <http://www.topquadrant.com/>'s
TopBraid Composer
<http://www.topquadrant.com/products/TB_Composer.html> (a graphical
development environment for modelling data) and TopBraid Live
<http://www.topquadrant.com/products/TB_Live.html> (an enterprise
SOA-capable Semantic Web application platform). Contact the OWLIM
team directly <mailto:[email protected]> for details of how to
obtain the OWLIM plug-in.
* All *control queries now use SPARQL Update syntax* (used mostly to
control the Lucene-based full-text search, RDF Rank and geo-spatial
plug-ins). This has a number of advantages, namely:
o No special control query pseduo-graph is required by the
Replication Cluster master in order to identify control queries
that must be pushed to all worker nodes
o SPARQL Updates use the corresponding SPARQL update protocol, so
they can be automatically processed by load-balancers that
examine URL patterns
o It is more consistent with the SPARQL language, since these
'control queries' cause a change of state in OWLIM
* *Incremental Lucene-based full-text search index* for updating the
index for specific resources or all un-indexed resources. Using this
technique can avoid the more expensive approach of rebuilding the
whole index frequently.
* *Incremental RDF Rank* allows the RDF rank for specific resources to
be (re-)computed as directed by the user. This technique can avoid
the more expensive approach of rebuilding all RDF Rank values
frequently.
* As well as the cache/index statistics, *performance analysis data*
is now provided about currently executing queries including: how
many results have been returned so far, how long it has been
executing, average time to return each result, etc.
* The *getting started* application has been restructured so that it
now works with remote repositories.
*Known problems with OWLIM 5.0*
* The behaviour of the 'include inferred' checkbox in the Sesame
Workbench is unpredictable when using OWLIM repositories.
* This version of OWLIM is *not backwardly compatible* with any
previous version. This means that images created with OWLIM 4.3 and
before will not work correctly with OWLIM 5.0 and must be
re-created. There have been a great many modifications to the
storage files, indexing structures, etc, and upgrade mechanisms have
proven too complex and probably slower than re-loading the database
anyway. Please *do not attempt to upgrade to OWLIM 5.0 unless you
drop and recreate all databases*. A migration tool, which allows for
automated re-loading of data from any Sesame-accessible repository,
is provided to ease the transition.
For further technical information and references to resolved technical
issues, please refer to the Release notes
<http://owlim.ontotext.com/display/OWLIMv50/OWLIM-SE+Release+notes> of
the corresponding edition of OWLIM. Full documentation for all OWLIM
editions is available online <http://owlim.ontotext.com> (click on the
OWLIM 5.0 link on the left hand side).
One can request further information and evaluation licences for OWLIM
from here <http://www.ontotext.com/owlim#download>.
The OWLIM team
April 2012
