Hi Steve,
QueryExecutionFactory lets me create a query on either a Model or a
Dataset.What is the difference?Which one does ARQ actually operate
on?Will ARQ create a Dataset and DataGraph if given (say) an in-memory
Model to be queried?
All SPARQL execution is on a dataset (actually a DatasetGraph, lower
level than Dataset which maps to Models, Resources from the building
blocks of Graphs and DatasetGraph).
Model is a presentation API - it has no state and the RDF is stored in
Graphs or DatasetGraphs.
Execution is of the algebra using class OpExecutor.
The algebra is extended to have quads and blocks of quads. OpExecutor
has both triple/basic graph pattern and quad/quadpattern steps. TDB
rewrite the algebra to quad form then executes that.
So actually touching the data comes down to "execute OpQuadPattern" or
"execute OpBGP" in OpExecutor. TDB executes QuadPattern natively; it has
its own subclass of OpExecutor. (TDB also has an adapter to execute over
a single graph as well.)
If executing in triple form for an in-memory, general purpose dataset, a
collection of graphs. The "execute OpBGP" step calls a rgistered
StageGenerator, which is an interface and can be set per-dataset.
StageGenerator receives a multi-triple BGP and returns the solutions -
get to see the whole BGP.
(Currently so does TIM, the native in-memory DatasetGraph even though it
is a quad store).
To extend for a custom graph implementation:
* do nothing - the default StageGenerator will turn the BGP execution
into graph.find calls.
if the graph can do better for a BGP:
* Provide your own StageGenerator.
and if a dataset concerned with quads or you want to see more of the
execution beyond BGPs:
* Provide your own OpExecutor subclass.
Everything maps back and forth between triples and quads so full SPARQL
functionality is available uniformly, but maybe not as efficiently as
possible.
When/where are indexes created and used?
Indexes are a feature of the storage, not the general execution strategy.
optimizing the SPARQL algebra
Optimziation has two parts: rewriting the algebra into "better" algebra.
This may include moving filters about.
Reordering BPs/QuadPatterns is done at the Storage or the default
StageGenerator.
Andy
On 05/04/2021 16:29, Steve Vestal wrote:
Thanks for the pointers to the excellent tutorial materials on how
SPARQL queries are processed and how various things affect
performance.After a little bit of digging and experimenting with ARQ,
I’m not clear on how things go from an optimized SPARQL algebra
expression to evaluation of leaf BGPs and when/where indexes are
created and used.
I assumed ARQ uses the Graph interface to access an underlying Model,
which can be backed by any of an in-memory model, a TDB, my own class
that implements the Graph interface, or an inference or union model
backed by any of these.The Graph interface does not have a “find”
method that accepts a multi-triple BGP as an input (as one of the
tutorials described), just finds for single-triple patterns.
QueryExecutionFactory lets me create a query on either a Model or a
Dataset.What is the difference?Which one does ARQ actually operate
on?Will ARQ create a Dataset and DataGraph if given (say) an in-memory
Model to be queried?Graph, Dataset, and DatasetGraph all support only
single-triple query patterns.The TDB documentation talks about
optimizing the SPARQL algebra, but it is the ARQ API that has
optimization configuration options.Some initial experiments with a
couple of ARQ Context settings resulted in little impact on a test
query issued to a series of increasingly larger in-memory
models.When/where are indexes created and used?
Thanks for any insight.
On 3/18/2021 9:45 AM, Steve Vestal wrote:
Thanks. I'm looking to get smarter in general about formulating
queries, particularly those with non-trivial graph structure, e.g.,
more than just a shallow tree of properties rooted in one resource,
maybe dags, maybe with cycles. I am open to post-processing query
results. (I do that already, generating and post-processing queries
are steps in the overall algorithm.)
On 3/18/2021 9:19 AM, Andy Seaborne wrote:
On 17/03/2021 22:45, Steve Vestal wrote:
I'd like to dig a bit deeper into SPARQL query performance, better
understand how different query formulations affect that, how ARQ
configuration parameters might be used to tune that. Can anyone
recommend a place to start reading beyond the SPARQL book and
language definition?
Hi Steve,
It's a bit "it depends on the query.
There was a presentation recently and while its not about ARQ, the
fundamental point that getting the basic graph pattern matching
working efficiently applies.
http://www.lotico.com/index.php/SPARQL_Query_Optimization_with_Pavel_Klinov
Do you have specific queries in mind or is this a general enquiry?
Andy
