| Magnus added a comment. |
Thanks @Smalyshev Yes these queries are outliers, but there is a good reason to run them (in this case, find images for missing items), and just by Wikidata growth, the number of outliers (queries that timeout) will increase, to the point where they are no longer outliers. Technical improvements (improved software, better hardware) can buffer some of this, but not all. As you show, some queries can be re-formulated (though the reason why one works but not the other remains a mystery to most users), but again, not forever.
As for my preferences for implementation:
- OAuth-regulated access point - rather not. It complicates script based queries. Can work around that, but...
- Separate service - yes, though ideally I'd just use the same URL and add a keyword to the SPARQL (we used to have /* SLOW_OK */ on the original toolserver SQL queries), which would then use a separate server in the background, and be transparent for the end user
- Offlining - if I could submit my query, get a token back, and then can query the token once every 5 min or so if it's done, and get the result back then, that would be fine with me
- What I do with the 1.8M items is I run full-text searches on them on Commons and Flickr, in this case. I am perfectly happy to cache the result myself for days or weeks and work on that list, but I need to get the list in the first place. LIMIT and OFFSET do not work for these when the full query doesn't, I tried
- For my own purposes, we could even have a Wikidata page with "named queries" (heading with the name, paragraph with the query, or something) that gets run on a regular basis; offlining, with a twist. This could produce static JSON files (maybe compressed) somewhere, under the query name. Additionally, the query service could offer returning the data via a fake query ("# NAMED QUERY Bob" etc), but that's optional, though it would have the benefit of using the same interface
- Transparent offlining - queries with a keyword get run with increased timeout (say, 10min), but only one of them at a time; the connection stays open though, and eventually returns, even if it takes hours. Feels dirty, though :-(
TASK DETAIL
EMAIL PREFERENCES
To: Magnus
Cc: chasemp, Lydia_Pintscher, Magnus, MichaelSchoenitzer_WMDE, MisterSynergy, doctaxon, Jonas, Ash_Crow, Daniel_Mietchen, Lucas_Werkmeister_WMDE, Jane023, Base, Gehel, Smalyshev, Ijon, Aklapper, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
Cc: chasemp, Lydia_Pintscher, Magnus, MichaelSchoenitzer_WMDE, MisterSynergy, doctaxon, Jonas, Ash_Crow, Daniel_Mietchen, Lucas_Werkmeister_WMDE, Jane023, Base, Gehel, Smalyshev, Ijon, Aklapper, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
