Re: Achieving reasonably performing federated queries

[Sarven Capadisli]

On 07/25/2013 11:49 AM, Rob Vesse wrote:

Can you provide examples of the query plan (the algebra) with the
optimizer on and off?

The issue is likely down to ARQs index join linearization optimization,
this works great for local data and small federated queries but can work
poorly for large federated queries.  If this is the case then the
optimized algebra will include sequence/conditional operators while the
unoptimized form will include join/leftjoin instead.

In current releases of ARQ this optimization is always on, in the
2.10.2-SNAPSHOTs you can now do the following to disable this specific
optimization:

ARQ.getContext().set(ARQ.optIndexJoinStrategy, false);

This will allow you to get the benefit of all the other optimizations that
do apply while ignoring the one that is likely causing the problem

As far as the joining goes, I'm not experiencing an improvement over Optimizer.noOptimizer(). Perhaps this is expected. I am however going to use it from now on. Thanks!

Poorly performing federated queries is not limited to ARQ or SPARQL but is
rather a general problem of service federation.  If federated services are
the bottleneck in your application then maybe you need to consider
re-architecting to bring some data locally.

I do cache the results.







-Sarven

Looking at the query times again, it looks like my numbers from earlier are way off because I was using the default TDB engine when I wrote the email. I will focus more on the application end because the total query to result time is not as bad as I originally thought. Is there a documentation on enabling or disabling different engines? Or which options are available for the command-line tools?

smime.p7s
Description: S/MIME Cryptographic Signature