[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata

[Sjoerddebruin]

Sjoerddebruin added a comment.


  So what is the next step? Do we need to discuss this step with some people 
for approval? The code for this should be minimal and it's a huge 
quality-of-life improvement for editors and bot maintainers...

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Sjoerddebruin
Cc: Sjoerddebruin, A_smart_kitten, Azertus, M2k_dewiki, Oudedutchman, Manuel, 
MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, matej_suchanek, 
Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, ArthurPSmith, 
Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Danny_Benjafield_WMDE, S8321414, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Dringsim, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, KimKelting, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata

[matej_suchanek]

matej_suchanek renamed this task from "In some cases, moving or deleting pages 
on a client wiki does not result in sitelink updates / removal on Wikidata." to 
"In some cases, moving or deleting pages on a client wiki does not result in 
sitelink updates / removal on Wikidata".

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: matej_suchanek
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[hoo]

hoo added a comment.


  In T143486#9679948 , 
@Lydia_Pintscher wrote:
  
  > In T143486#9679789 , 
@hoo wrote:
  >
  >> As far as I remember we deliberately chose not to auto-create users back 
then initially implementing this. I don't think this would be very hard to add 
this (in a nice way), but I haven't checked.
  >
  > Do you by chance remember why? This seems like a sensible thing to do at 
first sight to me.
  
  I think this was just a very conservative choice back then (also to make sure 
that libel account names don't propagate). But this was way back (2014ish), 
even before the SUL-finalization, so I don't think this still stands.
  
  If we end up implementing this: I just looked this up and 
`MediaWikiServices::getInstance()->getAuthManager()->autoCreateUser` should do 
what we need.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[Lydia_Pintscher]

Lydia_Pintscher added a comment.


  In T143486#9679789 , @hoo 
wrote:
  
  > As far as I remember we deliberately chose not to auto-create users back 
then initially implementing this. I don't think this would be very hard to add 
this (in a nice way), but I haven't checked.
  
  Do you by chance remember why? This seems like a sensible thing to do at 
first sight to me.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[hoo]

hoo added a comment.


  In T143486#9679761 , 
@matej_suchanek wrote:
  
  > Has the option to automatically create the user account prior to the update 
if it doesn't exist on Wikidata been actually considered? Is it impossible, or 
is it just unclear whether we want to do it?
  
  As far as I remember we deliberately chose not to auto-create users back then 
initially implementing this. I don't think this would be very hard to add this 
(in a nice way), but I haven't checked.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[M2k_dewiki]

M2k_dewiki added a comment.


  In T143486#9679761 , 
@matej_suchanek wrote:
  
  > Has the option to automatically create the user account prior to the update 
if it doesn't exist on Wikidata been actually considered? Is it impossible, or 
is it just unclear whether we want to do it?
  
  Also see
  
  - 
https://www.wikidata.org/wiki/Wikidata:Administrators%27_noticeboard#Proposal_for_admin_bot_to_create_local_accounts_for_users_in_other_wikis
  - 
https://www.wikidata.org/w/index.php?title=Wikidata%3AAdministrators%27_noticeboard=2115884256=2115382614

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: M2k_dewiki
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[matej_suchanek]

matej_suchanek added a comment.


  Has the option to automatically create the user account prior to the update 
if it doesn't exist on Wikidata been actually considered? Is it impossible, or 
is it just unclear whether we want to do it?

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: matej_suchanek
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[Lydia_Pintscher]

Lydia_Pintscher added a project: Wikidata Sitelinks.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[hoo]

hoo added a comment.


  In T143486#9674958 , 
@M2k_dewiki wrote:
  
  > Also see
  >
  > - 
https://www.wikidata.org/wiki/Wikidata:Project_chat#(5000)_unconnected_television_season_articles_in_the_english_language_wikipedia_after_pages_have_been_moved
  > - 
https://www.wikidata.org/w/index.php?title=Wikidata%3AProject_chat=2115538365=2114990113
  
  
  
  In T143486#9675122 , 
@M2k_dewiki wrote:
  
  > Also see
  >
  > - 
https://en.wikipedia.org/wiki/Wikipedia:Bots/Noticeboard#WikiData_discussion
  > - 
https://en.wikipedia.org/w/index.php?title=Wikipedia%3ABots%2FNoticeboard=1216561469=1216352889
  
  In this case the account performing the moves (Qwerfjkl_(bot) 
) doesn't 
have a Wikidata account, that's why these moves weren't propagated.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hoo
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[M2k_dewiki]

M2k_dewiki added a comment.


  Also see
  
  - 
https://en.wikipedia.org/wiki/Wikipedia:Bots/Noticeboard#Speedily-approved_page-moving_bot??
  - 
https://en.wikipedia.org/w/index.php?title=Wikipedia%3ABots%2FNoticeboard=1216561469=1216352889

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: M2k_dewiki
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[M2k_dewiki]

M2k_dewiki added a comment.


  Also see
  
  - 
https://www.wikidata.org/wiki/Wikidata:Project_chat#(5000)_unconnected_television_season_articles_in_the_english_language_wikipedia_after_pages_have_been_moved
  - 
https://www.wikidata.org/w/index.php?title=Wikidata%3AProject_chat=2115538365=2114990113

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: M2k_dewiki
Cc: M2k_dewiki, Oudedutchman, Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, 
abian, Nikki, matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, 
PokestarFan, ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, 
Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, karapayneWMDE, Invadibot, 
maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, 
QZanden, KimKelting, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[MisterSynergy]

MisterSynergy added a comment.


  Another status update:
  
  I have now migrated this job from PAWS to Toolforge (`msynbot` tool account) 
. Due to memory restrictions on Toolforge, I had to rewrite much of the code 
unfortunately. The memory-intensive operation is no longer done with 
Python/pandas; instead I use a temporary tool database so that the operations 
runs on database servers that are not subject to k8s memory limits. After 
several test runs, I am confident that there is no memory issue to be expected 
in the foreseeable future even with the largest wikis.
  
  There is now a weekly k8s-cronjob that should keep the backlog short. I am 
also continueing to log edits done by the bot so that I can provide some 
insight into the situations that lead to inexistent sitelinks on item pages if 
necessary.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[MisterSynergy]

MisterSynergy added a comment.


  Status update: the backlog of sitelinks to inexistent pages is cleared, 
except for:
  
  - Sitelinks to wikis that have been closed (their status is undetermined 
anyways; number of cases is unknown)
  - Sitelinks to Special pages, which appear as inexistent in some contexts but 
actually exist (these should not happen per guidelines, but there are ~1000 of 
such sitelinks currently in Wikidata)
  - Sitelinks to User pages where the user has a genered namespace prefix on 
the client wiki; these pages appear as inexistent in some scenarios as well; 
~10 cases)
  
  I do not plan to touch these at the moment.
  
  Besides that, I was able to clear the backlog with a custom script, except 
for ~75 really obscure cases which needed manual intervention. This means that 
my bot script is able to deal with almost everything that has shown up in the 
past.
  
  The statistics provided by me on July 12 above in this task is still valid. 
The main culprit to my experience are rate-limit issues when pages are deleted 
on the client wiki at a high rate (admin bot, Special:Nuke, i.e. not 
ratelimited) so that the sitelink removal on Wikidata cannot keep up. Since 
almost everything can be fixed automatically, I do not see an urgent need to 
change anything in the software.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[Manuel]

Manuel added a comment.


  Thank you for the update and for the great work, @MisterSynergy! \o/

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[MisterSynergy]

MisterSynergy added a comment.


  Status update: In the past days, I have removed deleted sitelinks for the 
"easy" cases where the reason is relatively obivous. This has reduced the 
number of open cases from ~60k to ~6k (i.e. 90% reduction). Findings:
  
  - Around 6k cases resulted from "move without redirect" scenarios on client 
wikis. This is much less than what I anticipated earlier, yet still a 
substantial amount.
  - Around 40k cases resulted from scenarios where the user batch-deleted 
plenty of pages on the client wiki at a high rate, either by using Special:Nuke 
or a custom deletion bot script. Since admins on client wikis usually enjoy 
noratelimit priviledges on the client wiki but not on Wikidata, this causes 
ratelimit issues when removing the sitelinks from Wikidata items. Since this is 
by far the most important reason why a deleted page might remain as a sitelink 
on the Wikidata item, it might be valuable to consider optimizations for this 
scenario.
  - Another 8k "deleted sitelinks" where not actually deleted, but their 
namespaces where renamed (on srwikinews and lmowiki only). I have simply 
updated the sitelinks so that this is not an issue any longer. There are more 
such cases waiting for a fix within the remaining 6k cases.
  
  Within the next days, I will have a look at the remaining "deleted sitelinks" 
in order to fix them as well. I will also set up a bot task that executes 
regularly, in order to keep the backlog short.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[Manuel]

Manuel added a comment.


  Yes, you are right @MisterSynergy! I am glad about your repair efforts and 
will stand down for now. Please let me know if we can contribute something from 
our side or if you have new insights from your investigations. Thank you, and 
best of luck with this!
  
  Also, thx for the additional info @Mike_Peel!

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[Mike_Peel]

Mike_Peel added a comment.


  Just to note that Pi bot was removing some of these links in 2018, focused on 
tgwiki, but I haven't been running that script recently. Documentation at 
https://www.wikidata.org/wiki/Wikidata:Requests_for_permissions/Bot/Pi_bot_9 
and code at https://github.com/mpeel/wikicode/blob/master/check_tgwiki.py . I'm 
happy to help with similar bot work if you need, but suspect you've got it well 
in hand @MisterSynergy!

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mike_Peel
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[MisterSynergy]

MisterSynergy added a comment.


  I don't think "User:Hoo bot" has much influence here as this bot has not 
edited Wikidata since 2016-10. While many cases are a couple of years old, they 
are not *that* old in fact. As much as I am aware, nobody has taken care of 
this for a long time now (but I am determined to do so…)

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[Manuel]

Manuel added a comment.


  Thank you so much @MisterSynergy, you are amazing! \o/
  
  Your analysis helps a lot! I wonder, if "User:Hoo Bot" might skew the 
analysis or not. @hoo, could you please give a little detail of what cases the 
bot is fixing and whether that changes MisterSynergy's analysis?
  
  Also, I am wondering, if we also should contribute to this effort from the 
developer side or if it makes sense to rely on your bot(s) for now? Should we 
e.g. at least try to fix 1A (and maybe AA) server-side?
  
  Do you have any thoughts on this?

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[MisterSynergy]

MisterSynergy added a comment.


  @Manuel: I have looked into this again. As of now, I have this list of 
potential reasons for sitelink update failures:
  
  1. Sitelink configuration-related reasons
1. A page on the client is "moved without a redirect" to another namespace 
that is forbidden (?) at Wikidata (such as a page move from main namespace to 
"User" or "Draft" namespace, e.g. when the page is not fit for the main 
namespace).
2. A redirect page on the client is "moved without a redirect". Redirect 
sitelinks are not permitted, thus the sitelink cannot be updated.
  2. User-based reasons
1. The user performing a sitelink change on a client does not have a local 
account at Wikidata
2. The user performing a sitelink change on a client is not permitted to 
edit Wikidata due to a block
3. The user performing a sitelink change on a client exceeds their 
rate-limit at Wikidata (seems rather unlikely)
  3. Page-based reasons
1. The item page is protected to a level that the user performing a 
sitelink change on a client is not allowed to edit it
  4. Wikidata edit capacities limited
1. Wikidata editing was generally rate-limited when the client sitelink had 
been changed (e.g. due to high maxlag), and the user made several sitelink 
changes in a short time (this might be the case with some bots)
2. Wikidata was read-only when the client sitelink had been changed
  5. Project configuration-related reasons
1. There are a couple of cases where a namespace has been renamed on client 
Wikis, but the sitelinks to that namespace have not been updated on Wikidata. 
There seem to be auto-redirects in place, but technically the old titles do not 
exist any more
  
  Currently I see roughly 60.000 sitelinks that do not exist as a page on the 
client. My impression is that 1A is the major and dominant contributor here, 
and maybe 4A to some extent as well. I will soon start to repair 1A cases 
including some logging for future investigations. If the backlog is shorter, I 
think it should become easier to learn something about the other scenarios as 
well.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[MisterSynergy]

MisterSynergy added a comment.


  @Manuel:
  
  - I got a bot task approved that allows me to tidy these sitelinks up 
regularly (i.e. remove from the item if the page is inexistent on the client 
wiki). This itself can be considered a "dirty" solution to the problem, but 
clearly not the best one.
  - However, it has not been executed yet due to a lack of time for Wikidata on 
my side in recent months.
  - AFAIR, the main issue currently is that the evaluation workflow is kinda 
demanding regarding memory usage. During drafting the code on PAWS with its 3 
GB memory limit, I offloaded parts of the evaluation for larger wikis to my 
local machine which has sufficient memory available. For a fully automated 
deployment on Toolforge, this is of course not possible. Instead, there may 
even be stricter memory limits applying on Toolforge than on PAWS.
  - Why does it need so much memory? My approach queries "all pages per client 
wiki" (from the client's `page` table) and "all sitelinks in Wikidata" (from 
Wikidata's `wb_items_per_site` table) into separate Pandas DataFrames and 
subsequently looks for differences using Python. In other words: I avoid 
checking millions of cases individually by sitelink, and use a pretty quick 
per-client-wiki approach instead that requires me to hold all information for a 
given client wiki in memory.
  
  So, the code itself is pretty much ready-to-roll, but I need to find a place 
to run this fully automated. If you are interested, I can try to generate an 
updated list of cases for further inspection but it would be helpful to really 
understand your needs. Do you want to further evaluate this?

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[Manuel]

Manuel added a comment.


  Thank you so much for your research @MisterSynergy, this helped a lot! In 
case, by any chance, you still have your list of cases (and analysis), then 
please let me know.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T143486: In some cases, moving or deleting pages on a client wiki does not result in sitelink updates / removal on Wikidata.

[Manuel]

Manuel renamed this task from "[feature request]  remove sitelinks / update 
sitelinks  on Wikidata when pages are deleted/moved on client wikis (all 
users)" to "In some cases, moving or deleting pages on a client wiki does not 
result in sitelink updates / removal on Wikidata.".
Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T143486

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Manuel, MisterSynergy, Mike_Peel, Bencemac, ARR8, abian, Nikki, 
matej_suchanek, Lydia_Pintscher, NicoScribe, Ladsgroup, PokestarFan, 
ArthurPSmith, Liuxinyu970226, Izno, hoo, Aklapper, Esc3300, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, 
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org