[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

So9q added a comment. In T291903#7739739 , @Hannah_Bast wrote: > To add to this, the two-index approach has another rather beautiful property: > > 1. It is important to understand that real-time updates have an inherent price. An

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

So9q added a comment. Most of SPARQL 1.1 is now supported with few exceptions. QLever is looking more promising by the day. Very nice work! See https://github.com/ad-freiburg/qlever/wiki/Current-deviations-from-the-SPARQL-1.1-standard for details TASK DETAIL

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Gehel closed this task as "Resolved". Gehel claimed this task. Gehel added a comment. Evaluation is documented in https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/WDQS_backend_alternatives TASK DETAIL https://phabricator.wikimedia.org/T291903 EMAIL

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Hannah_Bast added a comment. To add to this, the two-index approach has another rather beautiful property: 1. It is important to understand that real-time updates have an inherent price. An engine that supports real-time updates can never be as fast as a read-only engine. But with the

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

So9q added a comment. @Hannah_Bast informed in the last WDQS scaling meeting that QLever could have 2 indexes to provide near-realtime queries. See https://github.com/ad-freiburg/qlever/wiki/QLever-support-for-SPARQL-1.1-Update TASK DETAIL https://phabricator.wikimedia.org/T291903 EMAIL

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Gehel triaged this task as "Medium" priority. TASK DETAIL https://phabricator.wikimedia.org/T291903 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: DD063520, Justin0x2004,

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

So9q added a comment. In T291903#7411375 , @Hannah_Bast wrote: >> Oh, ok. Could you give an example of a query that has no "highly selective triples" so I can test it on QLever vs. BG? > > Here is a relatively simple query without a

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Hannah_Bast added a comment. @DD063520: You find some details at https://github.com/ad-freiburg/qlever/blob/master/docs/quickstart.md . For the current Wikidata, indexing takes around 24 hours on a AMD Ryzen 9 5900X with 128 GB of RAM and cheap HDDs. Our goal is an indexing time of at

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

DD063520 added a comment. Hi Hannah, nice work! I have some questions about the hardware requirements could you tell me: 1. the machine specs that you need to index the data (I read somewhere it takes 24h) 2. the final index size for the current Wikidata dump? Merci

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Hannah_Bast added a comment. > Oh, ok. Could you give an example of a query that has no "highly selective triples" so I can test it on QLever vs. BG? Here is a relatively simple query without a highly selective triple. It asks for the 100 people with the most professions. It requires a

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

So9q added a comment. In T291903#7393813 , @Hannah_Bast wrote: > I have now revised QLever's Quickstart page: https://github.com/ad-freiburg/qlever > > It allows you to build the code, build an index for a given set of triples, and

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

So9q added a comment. In T291903#7382766 , @Hannah_Bast wrote: > 2. Subqueries and predicate paths are also supported. Where is it written that they are not? Oh, I copy pasted a query from Scholia that had the "WITH" syntax and got

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Hannah_Bast added a comment. @Justin0x2004 Thanks, Justin. QLever already supports something like named subqueries. You can simply have the same subquery in multiple places and it will be evaluated only once and for the other occurrences, the result will be reused. We don't yet support

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Justin0x2004 added a comment. > Subqueries and predicate paths are also supported. Where is it written that they are not? @Hannah_Bast I think they mean this feature of BG: https://github.com/blazegraph/database/wiki/NamedSubquery TASK DETAIL

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Hannah_Bast added a comment. I have now revised QLever's Quickstart page: https://github.com/ad-freiburg/qlever It allows you to build the code, build an index for a given set of triples, and start the engine with just a few copy operations. With two example datasets, one small (120

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Hannah_Bast added a comment. I will provide a detailed reply lated today, also to the other thread. Four things for now: 1. The basic SPARQL features are all supported already, including LIMIT, OFFSET, ORDER BY, GROUP BY, HAVING, COUNT, DISTINCT, SAMPLE, GROUP_CONCAT, FILTER, REGEX,

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T291903 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: So9q, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana,

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

So9q added a comment. Given the limitations many Scholia queries need BG to work. TASK DETAIL https://phabricator.wikimedia.org/T291903 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: So9q Cc: So9q, Aklapper, MPhamWMF, CBogen, Namenlos314, Gq86,

[Wikidata-bugs] [Maniphest] T291903: Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster

So9q renamed this task from "Evaluating QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster" to "Evaluate QLever as a time lagging SPARQL backend to offload the BlazeGraph cluster". TASK DETAIL https://phabricator.wikimedia.org/T291903 EMAIL PREFERENCES