Hi Hugh,
if I open the provided files with an editor, there is no error.
For example, profile_q2_federated contain:
Connected to OpenLink Virtuoso
Driver: 07.20.3215 OpenLink Virtuoso ODBC Driver
result
LONG VARCHAR
_______________________________________________________________________________
http://www.bibsonomy.org/bibtex/20001ccbac4a192c385dd1a813c33e4a4
130--147
http://www.bibsonomy.org/bibtex/2001726ad493878da37ce74c327b64142
451--464
http://www.bibsonomy.org/bibtex/2002a5c36b038d703388ea3133d1c1d64
125--144
http://www.bibsonomy.org/bibtex/2002b342f8686b42ca9386b7101e57d03
1966--1972
http://www.bibsonomy.org/bibtex/2002f844c77738f07d6aed27377358948
36--50
http://www.bibsonomy.org/bibtex/200583305f841e25137f18dc67f63e83a
571-573
http://www.bibsonomy.org/bibtex/200825babcdaa38b3fa7afef00705b74e
1553-1558
http://www.bibsonomy.org/bibtex/2008eea7c2709538ca3824513b797dd5f
319--355
http://www.bibsonomy.org/bibtex/20091a9f693bb98983ac76724ecfaa344
195--205
http://www.bibsonomy.org/bibtex/200b84d4fae9f972bd06d1648119d1ac8
203--223
http://www.bibsonomy.org/bibtex/200d0f3fa9f6a4ab1fc84973c9f6d03f7
10--12
http://www.bibsonomy.org/bibtex/200d246b67269ee8de5bdb900b766f229 92-97
http://www.bibsonomy.org/bibtex/200ecc41577170f3a59cd67882c309d0a 77-86
http://www.bibsonomy.org/bibtex/200ecc846b53f52079c34af53e8badcfa
4--10
http://www.bibsonomy.org/bibtex/200ef9f672b4e826261dcd9219d5d0507 82-83
http://www.bibsonomy.org/bibtex/200f4976a86726f22802bd39e93386926
238--246
http://www.bibsonomy.org/bibtex/2010ebc7f46486e290469eb950a20d04d
319--325
http://www.bibsonomy.org/bibtex/2011380714119b1a98da3addca3fe0e50 18
http://www.bibsonomy.org/bibtex/2012c49708658e54a9e61214be4bdc4b0 27--36
http://www.bibsonomy.org/bibtex/2012c49708658e54a9e61214be4bdc4b0 27–36
{
time 1.8e-07% fanout 1 input 1 rows
Precode:
0: vector := Call vector ()
5: vector := Call vector (<c b>, <c x>)
10: BReturn 0
Subquery 31
{
time 1.5e-07% fanout 1 input 1 rows
Precode:
0: __stub := := artm 1
4: BReturn 0
END Node
time 6.5e-08% fanout 0 input 1 rows
Subquery Select(__stub)
}
time 1e+02% fanout 1000 input 1 rows
Precode:
0: ws_endpoint := := artm <tag 182 flag 1 c
http://141.87.4.8:8891/sparql>
4: ws_params := := artm vector
8: qtext_template := := artm <c SELECT ?b ?x
WHERE { ?x <http://swrc.ontoware.org/ontology#pages> ?b . }>
12: qtext_posmap := := artm <tag 225 c >
16: param_row := := artm vector
20: expected_vars := := artm vector
24: proc_ctr := := artm 0
28: <none> := Call __reset_temp ( 140051501624464 )
33: <proc table> := Call DB.DBA.SPARQL_SINV_IMP (<tag 182 flag 1 c
http://141.87.4.8:8891/sparql>, vector, <c SELECT ?b ?x
WHERE { ?x <http://swrc.ontoware.org/ontology#pages> ?b . }>, <tag 225 c >,
vector, vector)
40: BReturn 0
Key from temp (proc_ctr, RSET)
After code:
0: aref := Call aref (RSET, 1 )
5: x := Call __id2in (aref)
10: aref := Call aref (RSET, 0 )
15: b := Call __ro2sq (aref)
20: BReturn 0
time 2.7e-08% fanout 0 input 1000 rows
Select (x, b)
}
4987453 msec 99% cpu, 1.40551e+06 rnd 200089 seq 0% same seg
0.0686584% same pg
144592 disk reads, 143623 read ahead, 0.445211% wait
Compilation: 2 msec 0 reads 0% read 0 messages 0% clw
5 Rows. -- 4987467 msec.
Best
Andy
Von: Hugh Williams [mailto:hwilli...@openlinksw.com]
Gesendet: Montag, 29. Februar 2016 17:05
An: Nolle, Andreas <no...@hs-albsig.de>
Cc: virtuoso-users@lists.sourceforge.net
Betreff: Re: [Virtuoso-users] infrequent errors on parallel querying
Hi Andreas,
How are you running the script for the query profiles as all scripts provided
contain errors of the form:
Last login: Mon Feb 29 16:01:14 on ttys006
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated ; exit;
Hughs-MBP:~ hwilliams$
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated ; exit;
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 1:
Connected: command not found
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 2:
Driver:: command not found
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 3: result:
command not found
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 4: LONG:
command not found
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 5:
_______________________________________________________________________________:
command not found
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 7:
http://www.bibsonomy.org/bibtex/20001ccbac4a192c385dd1a813c33e4a4: No such file
or directory
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 8:
http://www.bibsonomy.org/bibtex/2001726ad493878da37ce74c327b64142: No such file
or directory
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 9:
http://www.bibsonomy.org/bibtex/2002a5c36b038d703388ea3133d1c1d64: No such file
or directory
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 10:
http://www.bibsonomy.org/bibtex/2002b342f8686b42ca9386b7101e57d03: No such file
or directory
/Users/hwilliams/Downloads/analysis_files/profile_q2_federated: line 11:
http://www.bibsonomy.org/bibtex/2002f844c77738f07d6aed27377358948: No such file
or directory
So they have not executed correct, thus no profiles etc are available …
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 29 Feb 2016, at 14:34, Nolle, Andreas
<no...@hs-albsig.de<mailto:no...@hs-albsig.de>> wrote:
Hi Hugh,
again many thanks for your reply and working at the weekend!
I've created the index just on a copy of the original database for some tests...
So all indexes (STATISTICS DB.DBA.RDF_QUAD;) created on these various instances
are the same, since these are all default indexes.
If you are interested on the output of STATISTICS DB.DBA.RDF_QUAD; for each
Virtuoso instance, please see the zip file referenced below.
The triple counts of each instance is
- Virtuoso instance running at port 8891: 17765873 triples
- Virtuoso instance running at port 8893: 27897291 triples
- Virtuoso instance running at port 8895: 168888956 triples
- Virtuoso instance running at port 8899: 72372256 triples
Right, I've already changed the ini files according to your suggestions:
- removing ServerThreads
- reducing MaxClientConnections to 200
- reducing MaxQueryMem
- setting AdjustVectorSize to 0
For profiling I shrinked the test queries such that it is probably easier to
find out the problem. By doing so, I came to the assumption that not the
FILTERS causes the long query evaluation time, but the size of the result set.
If I run q1:
SELECT ?x ?b
WHERE {
SERVICE <http://141.87.4.8:8891/sparql> {
?x <http://swrc.ontoware.org/ontology#edition> ?b .
} .
}
at lets say Virtuoso instance running at port 8895 it will end up within 2
minutes. This is because the service call will return only 32562 results.
But if I run q2:
SELECT ?x ?b
WHERE {
SERVICE <http://141.87.4.8:8891/sparql> {
?x <http://swrc.ontoware.org/ontology#pages> ?b .
} .
}
where the service call returns 313421 results, the query evaluation on the same
instance (running at port 8895) will take something around 1 hour!
Do you have any explanation for that or any suggestion how such long evaluation
times can be fixed?
If you are still interested in explain and profile plans of those queries,
please find it at
https://www.dropbox.com/s/ein0hxhtgk85ci9/analysis_files.zip?dl=0.
Please notice that *_federated means the queries are like above and are
evaluated at the Virtuoso instance running at port 8895, and *_local means the
evaluation of these queries (without SERVICE) at the Virtuoso instance running
at port 8891.
Best regards
Andy
-----Ursprüngliche Nachricht-----
Von: Hugh Williams [mailto:hwilli...@openlinksw.com]
Gesendet: Sonntag, 28. Februar 2016 22:21
An: Nolle, Andreas <no...@hs-albsig.de<mailto:no...@hs-albsig.de>>
Cc:
virtuoso-users@lists.sourceforge.net<mailto:virtuoso-users@lists.sourceforge.net>
Betreff: Re: [Virtuoso-users] infrequent errors on parallel querying
Hi Andreas,
Generally there should be no create additional indexes as the 2 Full & 3
partial indexes have been found to be sufficient for most use cases, with only
one use case encountered where an addtional partial index was required as
detailed at:
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning#Index%20Scheme%20Selection
Thus what indexes (STATISTICS DB.DBA.RDF_QUAD;) are created on these various
instances you have are they all the same or do they have different indexes as
the more indexes the more large the database and hence more memory required for
hosting it. Also are the triple counts the same on all these instances ?
Did you make any of the other INI file changes in my previous email, certainly
“AdjustVectorSize” should be set to 0 ?
You should also consider profiling the queries to determine if the best quey
plan is being used, which can be done with the “profile()” function as detailed
at:
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksAanalyzingSPARQLQuery
http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#querylogging - Query
Logging
Note there is are also some INI file params that can be set to control query
optimisation ie plans as if a bad plan is being chosen then these options can
in some cases enable better plans to be chosen, see:
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtQueryOptDiagnostic
If you can provide query plans, database statistics then these can be analysed
to trying and determine the cause of long running queries …
It certainly would be interesting to see how the query plans and database stats
vary being the instance where the query runs in msecs and the other were it
runs in 40+mins ...
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net<mailto:Virtuoso-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
