Smalyshev added a comment.
@Ijon I appreciate the issue, but having unauthenticated endpoint with extended timeout kinda defeats the purpose of timeout, and adding authentication is not trivial, so it has to wait until I get to it. Unless somebody wants to contribute a patch :)TASK
Ijon added a comment.
Thanks for the insight on the difference between those two versions. That is indeed usable, for this particular count.
But to the more general request? We do need some way to work around the timeout, sometimes, without abusing our personally knowing @Smalyshev.TASK
Smalyshev added a comment.
OK, so this query is fast:
SELECT (COUNT(*) AS ?count)
WHERE {
?article wdt:P31 wd:Q13442814 .
}
and this is slow:
SELECT (COUNT(?aricle) AS ?count)
WHERE {
?article wdt:P31 wd:Q13442814 .
}
I think for @Ijon's case the former works as well. Also, this one:
SELECT
Smalyshev added a comment.
Another example of a query timing out without lots of output is this simple count of scholarly articles (millions)
I think this one should be easy to do by first fetching all types with something like http://tinyurl.com/y9pnxg82 and then running counts for each of the
Ijon added a comment.
Yes. Another example of a query timing out without lots of output is this simple count of scholarly articles (millions)
SELECT (COUNT(?article) AS ?count)
WHERE {
?article wdt:P31/wdt:P279* wd:Q13442814
}
I think such counts are very reasonable queries to occasionally want
Lucas_Werkmeister_WMDE added a comment.
In T179879#4041380, @MichaelSchoenitzer wrote:
Maybe you see any other tricks to improve this type of query?
Adding a query hint helps:
SELECT ?item ?sitelinks WHERE {
?item wdt:P31 wd:Q5;
wikibase:sitelinks ?sitelinks. hint:Prior
Smalyshev added a comment.
But what I was referring to is that while above examples produce huge amount of data in the output and can therefore never be significantly faster by definition, the example I gave does not give big amounts of data as output.
Yes, as you correctly noted, what matters is
MichaelSchoenitzer added a comment.
That's the reason why queries time out. There's no magic to it - queries that time out are those that require the engine to process a huge amount of data.
Well that's obvious. But what I was referring to is that while above examples produce huge amount of data
Smalyshev added a comment.
So far all the examples for queries that are timing out are queries that produce massive amount of data or are very complex
That's the reason why queries time out. There's no magic to it - queries that time out are those that require the engine to process a huge amount
MichaelSchoenitzer added a comment.
So far all the examples for queries that are timing out are queries that produce massive amount of data or are very complex. But a simple pattern of a timing-out query I see very often is the following:
SELECT ?item ?sitelinks WHERE {
?item wdt:P31 wd:Q…;
Smalyshev added a comment.
Separate service - yes, though ideally I'd just use the same URL
If it will be separate service, it will be separate URL. Having two distinct services under the same URL would not really work well with how our LVS is set up, unless we do some complex tricks with
doctaxon added a comment.
In T179879#4006226, @Lucas_Werkmeister_WMDE wrote:
Oh, and the outer query is trying to get all the labels of the place of birth, place of death, country of citizenship, etc., in all languages. That’s also a terrible idea.
Not in all languages, but in German or the
doctaxon added a comment.
?item wdt:P27 wd:Q183 was my mistake, these are the German women. I need the Swedish, sorry: ?item wdt:P27 wd:Q34TASK DETAILhttps://phabricator.wikimedia.org/T179879EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: doctaxonCc: chasemp,
doctaxon added a comment.
In T179879#4006181, @Lucas_Werkmeister_WMDE wrote:
It only returns some two thousand results, so the surrounding query shouldn’t be a problem either…
No, the "minus"-variation gives a timout, too.TASK DETAILhttps://phabricator.wikimedia.org/T179879EMAIL
Lucas_Werkmeister_WMDE added a comment.
@doctaxon use MINUS instead of FILTER NOT EXISTS, and stop killing the optimizer, and then %subquery runs in less than 5 seconds.
SELECT ?item ?sitelinks WHERE {
?item wdt:P27 wd:Q183;
wdt:P21 wd:Q6581072;
wdt:P31 wd:Q5;
doctaxon added a comment.
@Jonas
This is such a query that runs into a timeout. It should give all Swedish women that have no sitelink to dewiki with counting the sitelinks and listing some properties to each item, and is needed for a dewiki community project. I cannot limit it because of the
Magnus added a comment.
Thanks @Smalyshev Yes these queries are outliers, but there is a good reason to run them (in this case, find images for missing items), and just by Wikidata growth, the number of outliers (queries that timeout) will increase, to the point where they are no longer outliers.
Jonas added a comment.
I can see the timeout brings a lot of frustration.
And I think we can agree that bumping the limit every few months cannot be the solution.
As a first step I would propose to create a process where one can submit a query that times out, so we can better understand the
doctaxon added a comment.
Maybe 120s limit? I think, we have to test it, but how?TASK DETAILhttps://phabricator.wikimedia.org/T179879EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: doctaxonCc: chasemp, Lydia_Pintscher, Magnus, MichaelSchoenitzer_WMDE,
doctaxon added a comment.
@Smalyshev @Magnus
Thank you, Magnus, this was my opinion, but my English is not so good like yours, so ...
Let's discuss about Magnus' suggestion, it sounds really good.TASK DETAILhttps://phabricator.wikimedia.org/T179879EMAIL
Magnus added a comment.
Semantics and drama aside, I think the point is that some useful queries are already creaking, or failing entirely.
Species without image times out, and I'm not even requesting taxon name (and we all know to stay away from the SERVICE labels for long queries by now).
Smalyshev added a comment.
but you didn't explain the reason why not.
Because it is a shared service, and running queries with no limit on execution time will deny other users opportunity to use this service.
if we stand still, then SPARQL will be useless in near future
I do not think running
bd808 added a comment.
In T179879#3998830, @doctaxon wrote:
@bd808 @chasemp What is your meaning about this?
@Smalyshev and his team are in charge of the WDQS software and servers.TASK DETAILhttps://phabricator.wikimedia.org/T179879EMAIL
Magnus added a comment.
How about a separate server just for "heavy" queries? For everyone? If that goes down occasionally, hey, so what?
To be extra fancy, maybe trigger it with a SPARQL comment, "# timeout:10" or something; every query with such a comment goes to the separate server?TASK
Smalyshev added a comment.
Maybe we should get a SPARQL server that allows a longer timeout period.
The problem with it, along with obvious resource constraints (hardware costs money, supporting service instances costs money & needs people), if we have longer timeout this means we will have more
Aklapper added a comment.
@doctaxon: I usually do not assign tasks to developers as I am not a manager of some unlimited pool of developers. :) Many tasks remain open until someone volunteers to investigate and provide a patch. The associated project tags provide information and links about
doctaxon added a comment.
Okay, thank you for merging, but this task is very old, too.
@Aklapper Can you assign this task to anyone of WMF, who provides the SPARQL servers. I guess, that the timeout duration can be increased server-side.
Thank you
Martin ...TASK
Jane023 added a comment.
I have come to understand (mostly through trial and error, and then asking around) that to reduce the time on the query you need to start with the thing that has the least number of items in the group you want to query. As these "sub groups" get bigger and bigger, I need
Base added a comment.
While I support the request completely, the query linked can be optimised by using just one label language and dumping slow label service, I was able to get results for Wikipedia as big as German: http://tinyurl.com/ycbr7zq2TASK
29 matches
Mail list logo
