On 05/04/2021 17:39, Steve Vestal wrote:
Thanks. Are the following impressions sort of correct?
If I have a moderately sizeable RDF model (say over 1M triples), I am
likely to get faster queries by loading it into TDB rather than read it
into an in-memory model.
Hard to tell which si faster.
At 1M triples, TDB will very likely be running with everything cached so
it's "in-memory" for query.
Speed difference will come down to the fact it is different code in
places. Algorithmically, they are much the same (comparing TIM, the
transactional in-memory dataset vs TDB).
If you try it out, do report your findings.
If I interface to an external database using the Graph interface (I
assume this is much easier than writing my own StageGenerator or
OpExecutor), then I should pay particular attention to how triples are
ordered in a query or in BGPs in the resulting algebra.
Yes.
But the default StageGenerator is some light reordering then calling
Graph.find. See the code
For RDF-star:
StageGeneratorGenericStar
QueryIterBlockTriplesStar
but if no embedded triple terms these are calling the same as:
StageGeneratorGeneric
QueryIterBlockTriples
QueryIterTriplePattern
Does the default StageGenerator execute finds in the order they appear
in a BGP in the algebra?
StageGeneratorGeneric - no.
There is a simple, fixed reordering.
Order by most grounded terms (being less keen on rdf:type).
Is there any reason to pick TDB over TDB2?
Not really especially for query use cases.
Andy
On 4/5/2021 8:34 AM, Andy Seaborne wrote:
Hi Steve,
QueryExecutionFactory lets me create a query on either a Model or a
Dataset.What is the difference?Which one does ARQ actually operate
on?Will ARQ create a Dataset and DataGraph if given (say) an
in-memory Model to be queried?
All SPARQL execution is on a dataset (actually a DatasetGraph, lower
level than Dataset which maps to Models, Resources from the building
blocks of Graphs and DatasetGraph).
Model is a presentation API - it has no state and the RDF is stored in
Graphs or DatasetGraphs.
Execution is of the algebra using class OpExecutor.
The algebra is extended to have quads and blocks of quads. OpExecutor
has both triple/basic graph pattern and quad/quadpattern steps. TDB
rewrite the algebra to quad form then executes that.
So actually touching the data comes down to "execute OpQuadPattern" or
"execute OpBGP" in OpExecutor. TDB executes QuadPattern natively; it
has its own subclass of OpExecutor. (TDB also has an adapter to
execute over a single graph as well.)
If executing in triple form for an in-memory, general purpose dataset,
a collection of graphs. The "execute OpBGP" step calls a rgistered
StageGenerator, which is an interface and can be set per-dataset.
StageGenerator receives a multi-triple BGP and returns the solutions -
get to see the whole BGP.
(Currently so does TIM, the native in-memory DatasetGraph even though
it is a quad store).
To extend for a custom graph implementation:
* do nothing - the default StageGenerator will turn the BGP execution
into graph.find calls.
if the graph can do better for a BGP:
* Provide your own StageGenerator.
and if a dataset concerned with quads or you want to see more of the
execution beyond BGPs:
* Provide your own OpExecutor subclass.
Everything maps back and forth between triples and quads so full
SPARQL functionality is available uniformly, but maybe not as
efficiently as possible.
When/where are indexes created and used?
Indexes are a feature of the storage, not the general execution strategy.
optimizing the SPARQL algebra
Optimziation has two parts: rewriting the algebra into "better"
algebra. This may include moving filters about.
Reordering BPs/QuadPatterns is done at the Storage or the default
StageGenerator.
Andy
On 05/04/2021 16:29, Steve Vestal wrote:
Thanks for the pointers to the excellent tutorial materials on how
SPARQL queries are processed and how various things affect
performance.After a little bit of digging and experimenting with ARQ,
I’m not clear on how things go from an optimized SPARQL algebra
expression to evaluation of leaf BGPs and when/where indexes are
created and used.
I assumed ARQ uses the Graph interface to access an underlying Model,
which can be backed by any of an in-memory model, a TDB, my own class
that implements the Graph interface, or an inference or union model
backed by any of these.The Graph interface does not have a “find”
method that accepts a multi-triple BGP as an input (as one of the
tutorials described), just finds for single-triple patterns.
QueryExecutionFactory lets me create a query on either a Model or a
Dataset.What is the difference?Which one does ARQ actually operate
on?Will ARQ create a Dataset and DataGraph if given (say) an
in-memory Model to be queried?Graph, Dataset, and DatasetGraph all
support only single-triple query patterns.The TDB documentation talks
about optimizing the SPARQL algebra, but it is the ARQ API that has
optimization configuration options.Some initial experiments with a
couple of ARQ Context settings resulted in little impact on a test
query issued to a series of increasingly larger in-memory
models.When/where are indexes created and used?
Thanks for any insight.
On 3/18/2021 9:45 AM, Steve Vestal wrote:
Thanks. I'm looking to get smarter in general about formulating
queries, particularly those with non-trivial graph structure, e.g.,
more than just a shallow tree of properties rooted in one resource,
maybe dags, maybe with cycles. I am open to post-processing query
results. (I do that already, generating and post-processing queries
are steps in the overall algorithm.)
On 3/18/2021 9:19 AM, Andy Seaborne wrote:
On 17/03/2021 22:45, Steve Vestal wrote:
I'd like to dig a bit deeper into SPARQL query performance, better
understand how different query formulations affect that, how ARQ
configuration parameters might be used to tune that. Can anyone
recommend a place to start reading beyond the SPARQL book and
language definition?
Hi Steve,
It's a bit "it depends on the query.
There was a presentation recently and while its not about ARQ,
the
fundamental point that getting the basic graph pattern matching
working efficiently applies.
http://www.lotico.com/index.php/SPARQL_Query_Optimization_with_Pavel_Klinov
Do you have specific queries in mind or is this a general enquiry?
Andy
