[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

M2k_dewiki added a comment. Please also see https://www.wikidata.org/wiki/Wikidata:Project_chat#Unique_constraint_violation_with_deleted_object https://www.wikidata.org/w/index.php?title=Wikidata%3AProject_chat=revision=1475543373=1475352852 TASK DETAIL

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

dcausse added a comment. Mitigation for deletes will be made using a script that polls for the deletion log and resync the items, ref: https://people.wikimedia.org/~dcausse/wdqs_manual_deletes/ . TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

M2k_dewiki added a comment. Please also see https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team/Query_Service_and_search#Q104776498_deleted_but_still_on_WQS_(2021-02-14)

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Gehel added a parent task: T267927: Reload wikidata journal from fresh dumps. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: Mbch331, Skim, Strepon, Multichill, Zbyszko,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

dcausse added a comment. Thanks for the comments. Inconsistencies for edits prior to Jan 20 (time when the last fix was deployed) are //expected// and will be fixed by the reload. Inconsistency on Q104982840 is more troubling as the delete was done after this date. I'll re-open this

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Mbch331 added a comment. Q104982840 which was deleted January 27th 08:48 CET still shows up in my query for invalid `P2397` values TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

agray added a comment. I am still seeing a handful of inconsistencies stemming from the same period of edits. For example this edit to Q5540133 on 23 December has not propagated to all databases, and so

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: Mbch331, Skim, Strepon, Multichill, Zbyszko, RKemper, Epidosis, dcausse, Tagishsimon,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

dcausse merged a task: T272120: Deleted item still gets shown in WDQS query results. dcausse added a subscriber: Mbch331. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, dcausse Cc:

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board. dcausse added a comment. Tried to find more inconsistencies using the query provided by @Multichill (https://w.wiki/ugf) and could not spot any while it was very easy to find one

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

RKemper added a comment. Post deploy check: `wdqs-updater.service` is now running without the `--import-async` flag as expected: `ExecStart=/bin/bash /srv/deployment/wdqs/wdqs/runUpdate.sh -n wdq -- --kafka

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

RKemper added a comment. (Puppet run successful on wdqs nodes following deploy) TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper Cc: Skim, Strepon, Multichill, Zbyszko, RKemper,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

RKemper moved this task from In Progress to To Be Deployed on the Discovery-Search (Current work) board. RKemper added a comment. Deploying https://gerrit.wikimedia.org/r/656833 now TASK DETAIL https://phabricator.wikimedia.org/T267175 WORKBOARD

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 656833 **merged** by Ryan Kemper: [operations/puppet@production] [wdqs] disable async imports https://gerrit.wikimedia.org/r/656833 TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Multichill added a comment. In T267175#6754527 , @dcausse wrote: > Checked a couple of these inconsistencies and they appear to all be out of order in the kafka topics. I suggest to disable `async imports` as I believe it might be

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 656833 had a related patch set uploaded (by DCausse; owner: DCausse): [operations/puppet@production] [wdqs] disable async imports https://gerrit.wikimedia.org/r/656833 TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

dcausse added a comment. Checked a couple of these inconsistencies and they appear to all be out of order in the kafka topics. I suggest to disable `async imports` as I believe it might be possible cause of these inconsistencies. TASK DETAIL https://phabricator.wikimedia.org/T267175

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Multichill added a comment. @RKemper so what's the status of this? I see a lot of cases where the last edit didn't get processed so the data in SPARQL is not consistent. See https://w.wiki/ugf for some examples. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 654914 **merged** by Ryan Kemper: [operations/puppet@production] wdqs: default to using kafka for updates https://gerrit.wikimedia.org/r/654914 TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 654914 had a related patch set uploaded (by Ryan Kemper; owner: Ryan Kemper): [operations/puppet@production] wdqs: default to using kafka for updates https://gerrit.wikimedia.org/r/654914 TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 646632 **merged** by Ryan Kemper: [operations/puppet@production] Revert "wdqs: use RecentChanges API for updates on all WDQS servers" https://gerrit.wikimedia.org/r/646632 TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 646631 **merged** by Ryan Kemper: [operations/puppet@production] [wdqs] re-enable polling kafka for updates on wdqs1010 https://gerrit.wikimedia.org/r/646631 TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

dcausse merged a task: T270975: Some lexemes cannot be obtained by SPARQL query. dcausse added subscribers: Strepon, Skim. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, dcausse Cc:

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Multichill added a comment. I've observed quite a lot of inconsistencies over the past two weeks. I haven't looked very extensively at it, but I'm getting the impression that blocks of edits are missed and the timestamps are around a spike at

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

dcausse moved this task from Needs review to In Progress on the Discovery-Search (Current work) board. dcausse reassigned this task from Zbyszko to RKemper. dcausse added subscribers: RKemper, Zbyszko. dcausse added a comment. We will re-enable the kafka poller that was disabled for security

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 646632 had a related patch set uploaded (by DCausse; owner: DCausse): [operations/puppet@production] Revert "wdqs: use RecentChanges API for updates on all WDQS servers" https://gerrit.wikimedia.org/r/646632 TASK DETAIL

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 646631 had a related patch set uploaded (by DCausse; owner: DCausse): [operations/puppet@production] [wdqs] re-enable polling kafka for updates on wdqs1010 https://gerrit.wikimedia.org/r/646631 TASK DETAIL https://phabricator.wikimedia.org/T267175

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 645313 **abandoned** by ZPapierski: [wikidata/query/rdf@master] Add processing delay to RC Poller Reason: Adding the delay was deemed to risky of a workaround. https://gerrit.wikimedia.org/r/645313 TASK DETAIL

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Zbyszko added a comment. Few details on the issue: - For extensively modified entities (e.g. by bots) log show that information provided by RecentChanges API isn't always up to date. This can lead to lost revisions - if the last change was among the ones not yet provided by the API,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, gerritbot Cc: Epidosis, dcausse, Tagishsimon, Lydia_Pintscher, CBogen, Z_thomas, agray,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

gerritbot added a comment. Change 645313 had a related patch set uploaded (by ZPapierski; owner: ZPapierski): [wikidata/query/rdf@master] Add processing delay to RC Poller https://gerrit.wikimedia.org/r/645313 TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Zbyszko added a comment. I started to investigate the issue, but had to get back to some previous issue - I should have some update before the end of the week, though. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Lydia_Pintscher added a comment. In T267175#6647957 , @Tagishsimon wrote: > This issue is still occurring. We still get different answers depending on which report server we hit. That is clearly suboptimal. Could we please have an

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

dcausse merged a task: T268408: Query returns outdated results . dcausse added a subscriber: Epidosis. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, dcausse Cc: Epidosis, dcausse,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

dcausse merged a task: T267924: WDQS Updater (based on recent changes) missed some updates. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, dcausse Cc: dcausse, Tagishsimon,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Zbyszko claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: Tagishsimon, Lydia_Pintscher, CBogen, Z_thomas, agray, Gehel, Lucas_Werkmeister_WMDE, Aklapper, M2k_dewiki,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Tagishsimon added subscribers: Lydia_Pintscher, Tagishsimon. Tagishsimon added a comment. This issue is still occurring. We still get different answers depending on which report server we hit. That is clearly suboptimal. Could we please have an update on action being taken, and/or the

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

CBogen added a comment. Timeboxing investigation to 1 day. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: CBogen Cc: CBogen, Z_thomas, agray, Gehel, Lucas_Werkmeister_WMDE, Aklapper,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Z_thomas added a comment. The same problem with https://de.wikipedia.org/wiki/Benutzer:Z_thomas/DE-BB-Stra%C3%9Fen-Barnim and the connected sites. I created a lot of streets in Wikidata during the last days, but no changes in the listeria-list refering to streets in this area. TASK DETAIL

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

agray added a comment. I flagged up the other case mentioned. As an example of the issue - At the time of writing, https://w.wiki/kxw returns four items, three edited on 12 October and one on 27 October. All have removing Q99671172 (which is now no longer linked from any items) as their

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Gehel added a comment. We'll need to do some investigation to see what kind of underlying bug is hiding here. We are planning a full data reload anyway, which should solve this. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Gehel moved this task from All WDQS-related tasks to Current work on the Wikidata-Query-Service board. Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T267175 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Gehel triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Lucas_Werkmeister_WMDE, Aklapper, M2k_dewiki, CBogen, Akuckartz, Nandana, Namenlos314, Lahi,

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Lucas_Werkmeister_WMDE added a comment. Probably an issue with the query service updater, yes. You’re getting different results if you hit different backend servers. > - https://www.wikidata.org/wiki/Q100417069?action=purge A purge to some objects did not change anything

[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

Reedy added a project: Wikidata-Query-Service. Reedy updated the task description. Restricted Application added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T267175 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Reedy Cc: