[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

Yurik added a comment. Clarification: there is a graph design sandbox , which re-renders the graph on every change. In production, the data is pulled by graphoid service (rarely, as it is behind varnish cache), and by client browsers (when

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

Smalyshev added a comment. @GWicke the context is that Yuri is building an interface that allows Graphs to query SPARQL endpoint. Since running the query each time graph is displayed is too expensive, we want some intermediate caching store that would store the results, possibly for the time

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

GWicke added a comment. @smalyshev, I would need more context to usefully comment on this. In particular, I wonder if there are a small number of queries that get a lot of hits, and if those queries can be cached for long enough to result in worthwhile hit rates. When discussing a use case

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

Christopher added a comment. @smalyshev I completely agree with the concept of an intermediate service between the nanosparqlserver and the client. I think that this service should "broker" requests (based on an options configuration object), and eval whether a query is re-executed against

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

Lydia_Pintscher added a comment. Yes as Stas says we will need to make it possible (and obvious) for people to get up-to-date results for maintenance tasks, work lists for editathons and current events. TASK DETAIL https://phabricator.wikimedia.org/T126730 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

Yurik added a comment. @smalyshev, I think the question is what to make "default": - should query results be cached by default, and not cached when a certain parameter is given, or the other way around? - should that parameter be part of the query, or should it be a header? - do we want to

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

Smalyshev added a comment. My opinion: > should query results be cached by default, and not cached when a certain > parameter is given, or the other way around? Not cached by default. But that may depend of what "default" is :) And how it behaves in production - if we get too much load, I may

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

aude added a comment. we might want to track entity usage of whatever entities are used and that could be incorporated into cache invalidation (essentially it's arbitrary access) TASK DETAIL https://phabricator.wikimedia.org/T126730 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

hoo added a comment. In https://phabricator.wikimedia.org/T126730#2022524, @Jonas wrote: > I think it is quite obvious that we need at least some caching before the > Query entities are finally deployed, because Graph extension is not the only > possible source of DoS requests. I didn't

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

Ricordisamoa added a subscriber: Ricordisamoa. Ricordisamoa added a comment. If a good caching strategy can be made to work without Query entities, why not reuse it for Scribunto access?  TASK DETAIL https://phabricator.wikimedia.org/T126730 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

Jonas added a comment. I think it is quite obvious that we need at least some caching before the Query entities are finally deployed, because Graph extension is not the only possible source of DoS requests. There are a lot of use cases were we are fine with cached (old) results. We could make

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

hoo added a comment. In https://phabricator.wikimedia.org/T126730#2022680, @Ricordisamoa wrote: > If a good caching strategy can be made to work without Query entities, why > not reuse it for Scribunto access?  Inline queries have a potentially high maintenance cost and don't have a

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

Smalyshev added a comment. Hmm... I'm not sure I want to do this in generic case. Not all queries should be cached - in fact, there are plenty of queries that change and should **not** be cached, and that's the whole point - such as data quality queries that may drive bots, etc. On the other