[Wikidata-bugs] [Maniphest] T307869: Request for new search profile for Wikidata that boosts Items for languages

2022-06-29 Thread dcausse
dcausse added a comment.


  No new profiles should be created for other wikibase installation as most of 
the wikidata specific options are managed in wmf specific config, not Wikibase 
nor CirrusSearch so the new Lexeme creation page should behave exactly as 
before.
  All the fixes we had to make in CirrusSearch should not impact anything 
except if other Wikibase installations had tuned such broken settings (but I 
doubt since they were totally broken and ineffective)
  
  The bugfix that might affect Wikibase installations relying on 
CirrusSearch is:
  
  - Fixed the handling of the configuration variable `wgWBCSStatementBoost` 
which was ignored.
  
  @Lea_WMDE @Evelien_WMDE do you have a link to such problems with Elastic in 
wbcloud?

TASK DETAIL
  https://phabricator.wikimedia.org/T307869

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE, dcausse
Cc: Lea_WMDE, Evelien_WMDE, ItamarWMDE, dcausse, Lucas_Werkmeister_WMDE, 
MPhamWMF, Aklapper, Lydia_Pintscher, Hellket777, Astuthiodit_1, 
Nikospappas1312, 786, Biggs657, karapayneWMDE, Invadibot, Universal_Omega, 
maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, 
Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Mahir256, QZanden, 
EBjune, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Bodhisattwa, 
Neuronton, Scott_WUaS, Wikidata-bugs, aude, Gryllida, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T307869: Request for new search profile for Wikidata that boosts Items for languages

2022-06-28 Thread dcausse
dcausse added a comment.


  There is yet another problem (see patch above that should fix it). I'm sorry 
that deploying this profile is such a pain, it demonstrates a clear problem in 
the way we (the search team) deploy such features/profiles and I filed T311528 
<https://phabricator.wikimedia.org/T311528> to discuss and hopefully improve 
the situation.

TASK DETAIL
  https://phabricator.wikimedia.org/T307869

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE, dcausse
Cc: ItamarWMDE, dcausse, Lucas_Werkmeister_WMDE, MPhamWMF, Aklapper, 
Lydia_Pintscher, Hellket777, Astuthiodit_1, Nikospappas1312, 786, Biggs657, 
karapayneWMDE, Invadibot, Universal_Omega, maantietaja, Juan90264, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, Lewizho99, 
Maathavan, _jensen, rosalieper, Bodhisattwa, Neuronton, Scott_WUaS, 
Wikidata-bugs, aude, Gryllida, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T307869: Request for new search profile for Wikidata that boosts Items for languages

2022-06-27 Thread dcausse
dcausse added a comment.


  Sorry about that, there was yet another issue in the WikibaseCirrusSearch 
Hook that caused the config to be ignored and cause the language selector 
profile context to simply use exactly the same settings as the classic entity 
completion search.
  There was also a typo in mw-config fixed in one the attached patch.

TASK DETAIL
  https://phabricator.wikimedia.org/T307869

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE, dcausse
Cc: ItamarWMDE, dcausse, Lucas_Werkmeister_WMDE, MPhamWMF, Aklapper, 
Lydia_Pintscher, Astuthiodit_1, Nikospappas1312, 786, Biggs657, karapayneWMDE, 
Invadibot, Universal_Omega, maantietaja, Juan90264, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, Lewizho99, 
Maathavan, _jensen, rosalieper, Bodhisattwa, Neuronton, Scott_WUaS, 
Wikidata-bugs, aude, Gryllida, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T307869: Request for new search profile for Wikidata that boosts Items for languages

2022-06-23 Thread dcausse
dcausse added a comment.


  The above patch should fix the issue, I forgot that profile repositories must 
have have unique names, sorry about that!

TASK DETAIL
  https://phabricator.wikimedia.org/T307869

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE, dcausse
Cc: ItamarWMDE, dcausse, Lucas_Werkmeister_WMDE, MPhamWMF, Aklapper, 
Lydia_Pintscher, Astuthiodit_1, Nikospappas1312, 786, Biggs657, karapayneWMDE, 
Invadibot, Universal_Omega, maantietaja, Juan90264, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, Lewizho99, 
Maathavan, _jensen, rosalieper, Bodhisattwa, Neuronton, Scott_WUaS, 
Wikidata-bugs, aude, Gryllida, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T307869: Request for new search profile for Wikidata that boosts Items for languages

2022-06-01 Thread dcausse
dcausse added a comment.


  The patches above add few placeholder to allow tuning a custom profile meant 
to be use by the language selector on Special:NewLexeme:
  
  - 
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexemeCirrusSearch/+/801791/
 adds a new `profile context` named `lexeme_new_lexeme_prefix` and few config 
options to allow to configure it
  - https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/801793/ is 
an example patch configuring this `lexeme_new_lexeme_prefix` profile context (I 
took the first approach mentioned in the description)
  - 
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseCirrusSearch/+/801790/
 is still WIP but adds the possibility to switch the `profile context` from the 
EntitySearchElastic constructor
  
  The open question is how to pass a new URI param added to `wbsearchentities` 
meant to switch between profiles back to `EntitySearchElastic`.

TASK DETAIL
  https://phabricator.wikimedia.org/T307869

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: ItamarWMDE, dcausse, Lucas_Werkmeister_WMDE, MPhamWMF, Aklapper, 
Lydia_Pintscher, Fernandobacasegua34, Astuthiodit_1, Nikospappas1312, 786, 
Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, maantietaja, 
Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Bsandipan, GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, 
Lewizho99, Maathavan, _jensen, rosalieper, Bodhisattwa, Neuronton, Scott_WUaS, 
Wikidata-bugs, aude, Gryllida, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T268864: WikibaseCirrusSearch uses Elastica's Match class

2022-05-18 Thread dcausse
dcausse changed the status of subtask T271777: Bump rufin/elastica (and related 
libraries) to versions that support PHP 8.0 from Stalled to 
Open.

TASK DETAIL
  https://phabricator.wikimedia.org/T268864

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, Reedy, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, Wilmanbeno, CBogen, ItamarWMDE, Akuckartz, DannyS712, Nandana, 
Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Nikerabbit, 
Jdforrester-WMF, Addshore, MaxSem, Mbch331, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T268865: WikibaseLexemeCirrusSearch uses Elastica's Match class

2022-05-18 Thread dcausse
dcausse changed the status of subtask T271777: Bump rufin/elastica (and related 
libraries) to versions that support PHP 8.0 from Stalled to 
Open.

TASK DETAIL
  https://phabricator.wikimedia.org/T268865

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Reedy, dcausse
Cc: Reedy, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, 
Wilmanbeno, CBogen, ItamarWMDE, Akuckartz, DannyS712, Nandana, Lahi, Gq86, 
GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, _jensen, rosalieper, 
Bodhisattwa, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Nikerabbit, 
Jdforrester-WMF, Addshore, MaxSem, Mbch331, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T307635: Query service results are missing some variables on some servers

2022-05-09 Thread dcausse
dcausse added a comment.


  This is extremely weird and I suspect a serious blazegraph bug that causes 
this. I could not reproduce the problem at the moment running the python script 
provided but it might certainly happen again in the future.
  I'm not sure how to proceed here but perhaps capturing the full blazegraph 
response when it occurs might help?

TASK DETAIL
  https://phabricator.wikimedia.org/T307635

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, LucasWerkmeister, Nikki, Aklapper, Astuthiodit_1, karapayneWMDE, 
Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T306054: Upgrade deployment-wdqs01 host to Buster

2022-04-25 Thread dcausse
dcausse added a project: Discovery-Search (Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T306054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Lucas_Werkmeister_WMDE, Mathew.onipe, Aklapper, Majavah, 
Peachey88, Jdforrester-WMF, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, 
Namenlos314, Lahi, Gq86, Bsandipan, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Jay8g, Krenair
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T305983: query.wikidata.org/bigdata/ldf - Language string should include language tag

2022-04-25 Thread dcausse
dcausse moved this task from Incoming to For Later on the 
Wikidata-Query-Service board.
dcausse triaged this task as "Medium" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T305983

WORKBOARD
  https://phabricator.wikimedia.org/project/board/891/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, OMrkvonO, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T306054: Upgrade deployment-wdqs01 host to Buster

2022-04-14 Thread dcausse
dcausse added a comment.


  I can confirm, this host is not used.

TASK DETAIL
  https://phabricator.wikimedia.org/T306054

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Lucas_Werkmeister_WMDE, Mathew.onipe, Aklapper, Majavah, 
Peachey88, Jdforrester-WMF, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, 
Namenlos314, Lahi, Gq86, Bsandipan, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, 
Jay8g, Krenair
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T305818: Perform a data transfer to wdqs2004 & wdqs1004 to reclaim burnt allocators

2022-04-11 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  wdqs2004 & wdqs1004 lost their free allocators too quickly (known issue that 
pops up //randomly//). We should do a data transfer from a sane source to 
reclaim these before other machines are affected by the same problem.
  
  AC:
  
  - wdqs2004 & wdqs1004 have their number of free allocators around 232k like 
other "sane" nodes.

TASK DETAIL
  https://phabricator.wikimedia.org/T305818

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302189: Regularly purge orphaned sitelink, value and reference nodes

2022-04-05 Thread dcausse
dcausse added a comment.


  Reason is that this data //may// be referenced by other items and thus cannot 
be deleted blindly without asking blazegraph: //"is this data used by another 
item?"// which would be too costly to ask for every edit.
  Another approach is to reload blazegraph from the dumps at regular intervals 
(TBD: once, twice or four times a year).

TASK DETAIL
  https://phabricator.wikimedia.org/T302189

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: MPhamWMF, Bugreporter, dcausse, Aklapper, Yair_rand, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

2022-04-01 Thread dcausse
dcausse added a comment.


  Actually wdqs2007, wdqs2004 and wdqs2003 also triggered jvmquake, GC activity 
increased and wdqs2007 & wdqs2003 were unresponsive for a couple minutes. For 
wdqs2004 there are no visible blips in the various graph. I guess we should 
relax the settings a bit more.

TASK DETAIL
  https://phabricator.wikimedia.org/T293862

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, 
Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

2022-04-01 Thread dcausse
dcausse added a comment.


  With the settings we properly detected wdqs1006 going down for 30minutes at 
`2022-04-22T12:30:00` (this 2minutes after the first blip in the graph).
  Unfortunately there was a false positive wdqs1012 at `2022-04-22T10:00:00` as 
this machine was unavailable from 2 minutes only.
  Unsure if it's still too sensitive or if we can accept having a couple false 
positives.

TASK DETAIL
  https://phabricator.wikimedia.org/T293862

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, 
Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T304365: Add property predicates to WCQS

2022-04-01 Thread dcausse
dcausse moved this task from Incoming to Scaling on the Wikidata-Query-Service 
board.
dcausse added a comment.


  I agree that federation is adding a lot of //boiler plate// and inspecting 
the shape of the IRIs is very fragile. But merging multiple graphs into the 
same store for ease of use is going against the recent discussions we had 
around the future of the WDQS architecture, it is also a bit more complex than 
it seems within the current data flows. Nevertheless I think the concern you 
raise is very valid and should be taken into account while we figure out if 
splitting the graph and building on top SPARQL federation is something we have 
to pursue and/or if some sub-graph are very central that they'd better be 
replicated to all sub-graphs.

TASK DETAIL
  https://phabricator.wikimedia.org/T304365

WORKBOARD
  https://phabricator.wikimedia.org/project/board/891/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, WikidataFacts, Aklapper, Dominicbm, Astuthiodit_1, karapayneWMDE, 
Invadibot, GFontenelle_WMF, MPhamWMF, maantietaja, FRomeo_WMF, CBogen, 
ItamarWMDE, Nintendofan885, Akuckartz, Nandana, JKSTNK, Namenlos314, Lahi, 
Gq86, E1presidente, Ramsey-WMF, Cparle, SandraF_WMF, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, 
Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Susannaanas, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, 
aude, Tobias1984, Manybubbles, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Raymond, Steinsplitter, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-03-31 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: elukey, akosiaris, Gehel, RKemper, bking, toan, Addshore, JMeybohm, 
Michael, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-03-31 Thread dcausse
dcausse added a comment.


  Thanks for the quick answer! (response inline)
  
  In T301147#7821582 <https://phabricator.wikimedia.org/T301147#7821582>, 
@JMeybohm wrote:
  
  >> - If the above is not possible could we mitigate this problem by 
over-allocating resources (increase the number of replicas) to the deployment 
to increase the chances of proper recovery if this situation happens again?
  >
  > If that makes sense from your POV you could do that ofc. I can't speak on 
how problematic this situation was compared to the potential waste of resources 
another pod means. But if the current workload is already maxing out the 
capacity of the 6 replicas you have, maybe bumping that to 7 might be smart 
anyways to account for peaks?
  
  The additional PODs won't be used as a flink job does not automatically scale 
so it would be a pure waste of resources (2.5G of reserved mem per additional 
POD). It would help I guess to improve redundancy in this scenario only if k8s 
assigns every POD to a distinct machine, in which case even with a single 
machine misbehaving flink would have enough redundancy to allocate the job to 
the spare POD. If k8s does do allocation randomly or that there are not enough 
k8s worker nodes (1 spare POD in our case would mean spreading the PODs over 8 
different machines) then it's probably not worth the waste of resources.
  
  > In T301147#7821422 <https://phabricator.wikimedia.org/T301147#7821422>, 
@dcausse wrote:
  >
  >> @JMeybohm do you see any additional action items that would improve the 
resilience of k8s in such scenario?
  >
  > Unfortunately we don't have any data on what went wrong on the node. I 
think T277876 <https://phabricator.wikimedia.org/T277876> would be a step in 
the right direction but I also doubt it would have fully prevented this issue 
(ultimately I can't say).
  
  Thanks, I'm adding it to the ticket description as a possible improvement.

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: elukey, akosiaris, Gehel, RKemper, bking, toan, Addshore, JMeybohm, 
Michael, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-03-31 Thread dcausse
dcausse moved this task from Ready for Development to Needs review on the 
Discovery-Search (Current work) board.
dcausse added a comment.


  Tentatively moving this ticket to //needs review// as I'm not sure sure we 
can do much more from the search team perspective.
  I think the last point to discuss was to investigate the reasons why a single 
k8s node that misbehaves could make a deployment unstable.
  @JMeybohm do you see any additional action items that would improve the 
resilience of k8s in such scenario?

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: elukey, akosiaris, Gehel, RKemper, bking, toan, Addshore, JMeybohm, 
Michael, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T305068: Alert when flink does not have the number of expected task managers

2022-03-31 Thread dcausse
dcausse claimed this task.
dcausse moved this task from Incoming to Needs review on the Discovery-Search 
(Current work) board.

TASK DETAIL
  https://phabricator.wikimedia.org/T305068

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, Michael, JMeybohm, Addshore, toan, bking, RKemper, Gehel, 
akosiaris, elukey, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T209859: Wikidata autocomplete (wbsearchentities) results with score <= 0

2022-03-30 Thread dcausse
dcausse removed EJoseph as the assignee of this task.
dcausse added a subscriber: EJoseph.

TASK DETAIL
  https://phabricator.wikimedia.org/T209859

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: EJoseph, Liuxinyu970226, dcausse, Smalyshev, EBernhardson, Aklapper, 
Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T209859: Wikidata autocomplete (wbsearchentities) results with score <= 0

2022-03-30 Thread dcausse
dcausse moved this task from Wikibase Search to needs triage on the 
Discovery-Search board.
dcausse assigned this task to EJoseph.

TASK DETAIL
  https://phabricator.wikimedia.org/T209859

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1849/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EJoseph, dcausse
Cc: Liuxinyu970226, dcausse, Smalyshev, EBernhardson, Aklapper, Astuthiodit_1, 
karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

2022-03-30 Thread dcausse
dcausse added a project: Discovery-Search.

TASK DETAIL
  https://phabricator.wikimedia.org/T238751

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe, dcausse
Cc: karapayneWMDE, Lucas_Werkmeister_WMDE, Ladsgroup, Gehel, Jheald, Joe, 
Addshore, Aklapper, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, 
Invadibot, Lalamarie69, MPhamWMF, LSobanski, maantietaja, Juan90264, 
Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, 
Kent7301, joker88john, CucyNoiD, Nandana, jijiki, Klaas_Z4us_V, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, 
QZanden, EBjune, LawExplorer, Lewizho99, Maathavan, elukey, _jensen, 
rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Jay8g
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-03-30 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: elukey, akosiaris, Gehel, RKemper, bking, toan, Addshore, JMeybohm, 
Michael, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-03-30 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: elukey, akosiaris, Gehel, RKemper, bking, toan, Addshore, JMeybohm, 
Michael, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T305068: Alert when flink does not have the number of expected task managers

2022-03-30 Thread dcausse
dcausse created this task.
dcausse added projects: Wikidata-Query-Service, Wikidata, Discovery-Search 
(Current work).

TASK DESCRIPTION
  As a maintainer of a flink session cluster I want to be alerted when the 
number of taskmanagers is not what the deployment expects so that I can react 
quickly.
  
  It may happen that k8s is preferring to reboot containers on a broken k8s 
node rather than migrate the pod to a new pod (see parent ticket), for k8s this 
deployment may appear to be working properly but for flink the resources it 
expects are not available and the job it's supposed to run will remain in the 
SCHEDULED state.
  
  AC:
  
  - alert when the number of task managers is below a certain threshold

TASK DETAIL
  https://phabricator.wikimedia.org/T305068

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, Michael, JMeybohm, Addshore, toan, bking, RKemper, Gehel, 
akosiaris, elukey, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294076: Blazegraph and MariaDB contain different sitelinks at Wikidata

2022-03-29 Thread dcausse
dcausse moved this task from Waiting to Needs Reporting on the Discovery-Search 
(Current work) board.
dcausse added a comment.


  The reconciliation process is running and should auto-correct missed updates 
couple hours after they're performed.
  I also fixed the inconsistencies listed here and other related tickets. 
Please let me know if you still find errors.

TASK DETAIL
  https://phabricator.wikimedia.org/T294076

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Lucas_Werkmeister_WMDE, Sjoerddebruin, dcausse, William_Avery, RShigapov, 
Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, 
CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol

2022-03-29 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302494

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper, dcausse
Cc: bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, 
Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol

2022-03-29 Thread dcausse
dcausse moved this task from In Progress to Needs Reporting on the 
Discovery-Search (Current work) board.
dcausse added a comment.


  Moved remaining work in T304914 <https://phabricator.wikimedia.org/T304914>.

TASK DETAIL
  https://phabricator.wikimedia.org/T302494

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper, dcausse
Cc: bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, 
Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol

2022-03-29 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302494

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper, dcausse
Cc: bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, 
Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T304914: Remove the presto client for swift from the flink image

2022-03-29 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  As a maintainer of a flink session cluster I want to stop using the presto 
client for swift present in the flink image so that I can migrate to newer 
version of flink since it was removed.
  
  This is a followup of T302494 <https://phabricator.wikimedia.org/T302494> 
where we dropped this dependency from the jobs running in the flink session 
cluster. This task is about dropping this swift client from the image.
  
  Existing flink session clusters rely on this swift client to store their H/A 
related data (e.g. job jars). This means we must migrate existing clusters to 
using s3 as a simple drop-in replacement is unlikely to work.
  
  Suggested migration procedure:
  
  - For codfw
- route wdqs & wcqs to eqiad only
- adapt the wikidata maxlag to poll eqiad only
- stop (with a savepoint) all the jobs (WDQS & WCQS) running on the codfw 
k8s wikikube cluster
- undeploy all the k8s deployments under the `rdf-streaming-updater` 
namespace (dropping all flink generated configmaps might be necessary by e.g. 
recreating the k8s namespace)
- delete the flink_ha_storage folder on the corresponding s3 bucket 
(`rdf-streaming-updater-codfw`)
- drop presto-swift from 
https://gerrit.wikimedia.org/g/wikidata/query/flink-rdf-streaming-updater and 
create a new image
- adapt the patch generated by PipelineLib when merging the patch above and 
remove all mentions to swift from `deployment-charts` (possibly adapting 
existing patch: 
https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/766123)
- deploy the chart to the rdf-streaming-updater namespace in codfw (which 
should be empty)
- deploy the flink jobs (WCQS & WDQS) from their corresponding savepoints
- repool codfw & resume polling codfw for wikidata maxlag calculation
  - For eqiad (do all the above replacing eqiad with codfw and vice versa)
  
  Note that most of this procedure can be tested against the staging cluster 
(omitting the parts about the routing live traffic and wikidata maxlag)
  
  AC:
  
  - none of the flink session clusters are using the presto swift client

TASK DETAIL
  https://phabricator.wikimedia.org/T304914

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T242453: Deadlock in blazegraph blocking all queries and updates

2022-03-28 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T242453

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: RKemper, bking, RLazarus, Legoktm, Gehel, William_Avery, CDanis, Addshore, 
dcausse, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T242453: Deadlock in blazegraph blocking all queries and updates

2022-03-28 Thread dcausse
dcausse reopened this task as "Open".
dcausse added a comment.


  re-opening, seems to happen more frequently

TASK DETAIL
  https://phabricator.wikimedia.org/T242453

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: RLazarus, Legoktm, Gehel, William_Avery, CDanis, Addshore, dcausse, 
Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, 
CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T303256: WDQS servers should use skolem for wikibaseSomeValueMode

2022-03-15 Thread dcausse
dcausse moved this task from Needs review to Needs Reporting on the 
Discovery-Search (Current work) board.
dcausse added a comment.


  `wikibase:isSomeValue` is functioning properly again.

TASK DETAIL
  https://phabricator.wikimedia.org/T303256

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: seav, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302396: Investigate EOFException when performing the first checkpoint after restoring from a savepoint

2022-03-15 Thread dcausse
dcausse added a comment.


  S3 <https://phabricator.wikimedia.org/S3> is confirmed to have fixed this 
issue, all jobs are now running 0.3.104 of the streaming-updater and are using 
the s3 client to persist their durable state.

TASK DETAIL
  https://phabricator.wikimedia.org/T302396

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Gehel, EBernhardson, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, 
Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol

2022-03-15 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302494

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper, dcausse
Cc: bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, 
Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol

2022-03-15 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302494

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper, dcausse
Cc: bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, 
Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302830: query service: (Alert) Reduced availability for job jmx_wdqs_updater in eqiad

2022-03-14 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302830

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper, dcausse
Cc: Aklapper, RKemper, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T303256: WDQS servers should use skolem for wikibaseSomeValueMode

2022-03-14 Thread dcausse
dcausse added a comment.


  might be accidentally fixed by merging 
https://gerrit.wikimedia.org/r/c/operations/puppet/+/742670 (it's still unclear 
why it's broken in the first place)

TASK DETAIL
  https://phabricator.wikimedia.org/T303256

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: seav, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol

2022-03-14 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302494

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper, dcausse
Cc: Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, 
Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, 
Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293063: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes

2022-03-14 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T293063

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: JMeybohm, Jelto, Aklapper, jijiki, dcausse, Astuthiodit_1, Arnoldokoth, 
karapayneWMDE, Invadibot, MPhamWMF, GeminiAgaloos, maantietaja, wkandek, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Addshore, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293063: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes

2022-03-14 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T293063

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: JMeybohm, Jelto, Aklapper, jijiki, dcausse, Astuthiodit_1, Arnoldokoth, 
karapayneWMDE, Invadibot, MPhamWMF, GeminiAgaloos, maantietaja, wkandek, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Addshore, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T303256: WDQS servers should use skolem for wikibaseSomeValueMode

2022-03-08 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T303256

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T303256: WDQS servers should use skolem for wikibaseSomeValueMode

2022-03-08 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T303256

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T303256: WDQS servers should use skolem for wikibaseSomeValueMode

2022-03-08 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  The function `wikibase:isSomeValue` should filter skolem and not blank nodes 
on wdqs servers. It seems that this setting was recently acctidentaly dropped 
from the server configurations.
  
  AC:
  
  - blazegraph should be started with `-DwikibaseSomeValueMode=skolem`

TASK DETAIL
  https://phabricator.wikimedia.org/T303256

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

2022-03-07 Thread dcausse
dcausse added a comment.


  Pushed 
https://gitlab.wikimedia.org/repos/search-platform/jvmquake/-/merge_requests/1 
(up for review) to have a debian package that we could install on production 
machines.

TASK DETAIL
  https://phabricator.wikimedia.org/T293862

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol

2022-03-07 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302494

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper, dcausse
Cc: Aklapper, dcausse, Fernandobacasegua34, 786, Suran38, Biggs657, 
karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T279541: Add a reconciliation strategy to the wdqs streaming updater

2022-02-25 Thread dcausse
dcausse added a comment.


  Deployment of this feature has been stopped due to T302340 
<https://phabricator.wikimedia.org/T302340>.

TASK DETAIL
  https://phabricator.wikimedia.org/T279541

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: EBernhardson, RShigapov, dcausse, Aklapper, Fernandobacasegua34, 786, 
Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, 
maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294076: Blazegraph and MariaDB contain different sitelinks at Wikidata

2022-02-25 Thread dcausse
dcausse merged a task: T302458: Q108896181 keeps showing up as having zero 
statements.
dcausse added subscribers: Sjoerddebruin, Lucas_Werkmeister_WMDE.

TASK DETAIL
  https://phabricator.wikimedia.org/T294076

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Lucas_Werkmeister_WMDE, Sjoerddebruin, dcausse, William_Avery, RShigapov, 
Aklapper, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302458: Q108896181 keeps showing up as having zero statements

2022-02-25 Thread dcausse
dcausse closed this task as a duplicate of T294076: Blazegraph and MariaDB 
contain different sitelinks at Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T302458

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Lucas_Werkmeister_WMDE, Aklapper, Sjoerddebruin, karapayneWMDE, 
Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302458: Q108896181 keeps showing up as having zero statements

2022-02-25 Thread dcausse
dcausse added a comment.


  Thanks for the report, these inconsistencies are due to missed updates which 
will be automatically fixed once T279541 
<https://phabricator.wikimedia.org/T279541> is deployed.
  I'm marking this ticket as a duplicate of T294076 
<https://phabricator.wikimedia.org/T294076> since they share the same root 
cause and same fix.

TASK DETAIL
  https://phabricator.wikimedia.org/T302458

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Lucas_Werkmeister_WMDE, Aklapper, Sjoerddebruin, karapayneWMDE, 
Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302494: The WDQS Streaming Updater should use S3 to access thanos-swift instead of the native swift protocol

2022-02-24 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  Followup of T302396 <https://phabricator.wikimedia.org/T302396>.
  
  The thanos-swift cluster is S3 <https://phabricator.wikimedia.org/S3> 
compatible so we should use that instead of the native swift client which we 
customized to implement tmp auth and has been removed from the official flink 
distribution: https://issues.apache.org/jira/browse/FLINK-21819.
  
  AC:
  
  - WDQS Streaming Updater is thanos-swift through the s3 protocol

TASK DETAIL
  https://phabricator.wikimedia.org/T302494

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302396: Investigate EOFException when performing the first checkpoint after restoring from a savepoint

2022-02-23 Thread dcausse
dcausse added a comment.


  Root cause seems swift related:
  
  Saw this in taskmanager logs:
  
  `Received IOException while reading 
'swift://rdf-streaming-updater-codfw.thanos-swift/wikidata/savepoints/savepoint-0d1c37-86ed4cb29023/bc3ce8ed-70b2-4e91-a81b-07f585dd0f1f',
 attempting to reopen: 
org.apache.flink.fs.openstackhadoop.shaded.org.apache.hadoop.fs.swift.exceptions.SwiftConnectionClosedException:
 Connection to Swift service has been closed: read() -all data consumed`

TASK DETAIL
  https://phabricator.wikimedia.org/T302396

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: EBernhardson, Aklapper, dcausse, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302396: Investigate EOFException when performing the first checkpoint after restoring from a savepoint

2022-02-23 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302396

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302396: Investigate EOFException when performing the first checkpoint after restoring from a savepoint

2022-02-23 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T302396

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302396: Investigate EOFException when performing the first checkpoint after restoring from a savepoint

2022-02-23 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  While deploying a new version of the streaming-updater (0.3.103) flink failed 
with:
  
java.io.EOFException
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:170)
at 
org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer.deserialize(BytePrimitiveArraySerializer.java:82)
at 
org.apache.flink.contrib.streaming.state.restore.RocksDBFullRestoreOperation.restoreKVStateData(RocksDBFullRestoreOperation.java:229)
at 
org.apache.flink.contrib.streaming.state.restore.RocksDBFullRestoreOperation.restoreKeyGroupsInStateHandle(RocksDBFullRestoreOperation.java:158)
at 
org.apache.flink.contrib.streaming.state.restore.RocksDBFullRestoreOperation.restore(RocksDBFullRestoreOperation.java:142)
at 
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:284)
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:587)
at 
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:93)
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
at 
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
at 
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:425)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:535)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:525)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:565)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
at java.base/java.lang.Thread.run(Thread.java:834)
  
  notes:
  
  - the problem occurs when using two different savepoints:
- (thanos) 
`rdf-streaming-updater-codfw/commons/savepoints/deploy_0_3_103/savepoint-c4b021-9c4cd6541ec7/`
- (thanos) 
`rdf-streaming-updater-codfw/commons/savepoints/savepoint-c4b021-818dd669a47e/`
  - the exception is only visible on a taskmanager POD running on 
`kubernetes2003` (logs 
<https://logstash.wikimedia.org/app/discover#/?_g=(filters:!(),query:(language:lucene,query:kubernetes2003),refreshInterval:(pause:!t,value:0),time:(from:'2022-02-22T16:00:00.000Z',to:'2022-02-22T19:30:00.000Z'))&_a=(columns:!(log,host,message,kubernetes.pod_id),filters:!(),index:'logstash-*',interval:auto,query:(language:lucene,query:'kubernetes.master_url:%22https:%2F%2Fkubemaster.svc.codfw.wmnet%22%20AND%20kubernetes.namespace_name:%22rdf-streaming-updater%22%20AND%20java.io.EOFException%20AND%20kubernetes.labels.component:taskmanager'),sort:!())>)
  - the problem occurs when loading the savepoints using the previous version 
0.3.99
  - the deploy worked fine on staging for both wdqs and wcqs, fine as well on 
wcqs at codfw
  - the system was able to resume using 0.3.99 and a previous checkpoint 
`rdf-streaming-updater-codfw/wikidata/checkpoints/e245dd1e76d56d9ded351b27cf2d4c2a/chk-415014`.
  
  AC:
  
  - understand the root cause of the failure

TASK DETAIL
  https://phabricator.wikimedia.org/T302396

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T302189: Query service retains orphaned sitelinks

2022-02-21 Thread dcausse
dcausse closed this task as "Declined".
dcausse added a comment.


  Sitelink orphans are not clean up at update time for performance reasons, 
same is done for orphaned `values` and `references`. The database is cleaned up 
during a full reload which we try to plan once a year. Declining as this is 
done "on-purpose" but please feel free to re-open if this causes any usability 
issue on your side.

TASK DETAIL
  https://phabricator.wikimedia.org/T302189

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, Yair_rand, karapayneWMDE, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301695: Special:EntityData on test-commons.wikimedia.org produces wrong sdcdata prefix in its RDF output

2022-02-14 Thread dcausse
dcausse created this task.
dcausse added a project: SDC General.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  When I extract the RDF content using Special:EntityData of an image with a 
MediaInfo slot on test-commons-wikimedia.org I want it to be configured simarly 
than commons.wikimedia.org so that I can use test-commons as a test service.
  
  When calling 
https://test-commons.wikimedia.org/wiki/Special:EntityData/M410.ttl I see:
  
@prefix sdcdata: 
<https://test-commons.wikimedia.org/wiki/testcommons:Special:EntityData/> .
  
  But I expect:
  
@prefix sdcdata: 
<https://test-commons.wikimedia.org/wiki/Special:EntityData/> .

TASK DETAIL
  https://phabricator.wikimedia.org/T301695

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, GFontenelle_WMF, FRomeo_WMF, CBogen, Nintendofan885, 
JKSTNK, Lahi, E1presidente, Ramsey-WMF, Cparle, SandraF_WMF, Tramullas, Acer, 
Salgo60, Silverfish, Susannaanas, Jane023, Wikidata-bugs, Base, matthiasmullie, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Raymond, Steinsplitter
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-02-10 Thread dcausse
dcausse added a comment.


  In T301147#7692414 <https://phabricator.wikimedia.org/T301147#7692414>, 
@JMeybohm wrote:
  
  > In T301147#7689837 <https://phabricator.wikimedia.org/T301147#7689837>, 
@dcausse wrote:
  >
  >> @JMeybohm we're still investigating why the application did not properly 
recover while kubernetes1014 went down but if you have ideas on the two 
questions in the ticket description this would be very helpful, thanks!
  >
  > Unfortunately I'm not exactly sure what happened to the node. What I know 
is that the system load surged (potentially due to high iowait) on the system, 
leaving running processes practically starving but the system was still 
responding to ICMP and kubernetes status heartbeats still (mostly) worked. 
Leaving the node flipping between Ready/NotReady state.
  > That means the node was not actually down from k8s POV, which is why no new 
Pods where created until I drained the node respectively before I powercycled 
it (as evicting pods was actually hanging as well, as k8s tries to be nice and 
the node still was in it's overloaded state).
  
  Thanks! I've updated the task description with few action items, please let 
us know if you see something else we should do to improve this.

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: toan, Addshore, JMeybohm, Michael, Aklapper, dcausse, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-02-10 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: toan, Addshore, JMeybohm, Michael, Aklapper, dcausse, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

2022-02-09 Thread dcausse
dcausse claimed this task.
dcausse moved this task from Ready for Development to In Progress on the 
Discovery-Search (Current work) board.

TASK DETAIL
  https://phabricator.wikimedia.org/T293862

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-02-08 Thread dcausse
dcausse added a comment.


  k8s seems to have tried to kill the container for the whole period according 
messages like: Container flink-session-cluster-main-taskmanager failed liveness 
probe, will be restarted 
<https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-syslog-2022.02.07?id=C_LI0n4B-VMqbJtQBjqU>
 (searching for 
`k8s_event.involvedObject.uid:"1db45eb6-2405-4aa3-bec1-71fcdbbe4f9a"`).

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Addshore, JMeybohm, Michael, Aklapper, dcausse, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-02-07 Thread dcausse
dcausse added a subscriber: JMeybohm.
dcausse added a comment.


  @JMeybohm we're still investigating why the application did not properly 
recover while kubernetes1014 went down but if you have ideas on the two 
questions in the ticket description this would be very helpful, thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: JMeybohm, Michael, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T301147: The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)

2022-02-07 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  For 7 hours (`2022-02-06T23:00:00` to `2022-02-07T06:20:00`) the streaming 
updater in `eqiad` stopped working properly preventing edits to flow to all the 
wdqs machines in eqiad.
  The lag started to rise in eqiad and caused edits to be throttled during this 
period:
  
  F34944091: Capture d’écran du 2022-02-07 11-40-08.png 
<https://phabricator.wikimedia.org/F34944091>
  
  Investigations:
  
  - the streaming updater for WCQS went down from `2022-02-06T16:32:00` to 
`2022-02-06T23:00:00`
  - the streaming updater for WDQS went down from `2022-02-06T23:00:00` to 
`2022-02-07T06:20:00`
  - the number of total task slots went down to 20 from 24 (4tasks == 1pod) 
between `2022-02-06T16:32:00` and `2022-02-07T06:20:00` causing resource 
starvation and preventing both jobs from running at the same time 
(`flink_jobmanager_taskSlotsTotal{kubernetes_namespace="rdf-streaming-updater"}`)
  - kubernetes1014 (T301099 <https://phabricator.wikimedia.org/T301099>) seemed 
to have showed problems during this same period (`2022-02-06T16:32:00` to 
`2022-02-07T06:20:00`)
  - the deployment used by the updater used one POD 
(`1db45eb6-2405-4aa3-bec1-71fcdbbe4f9a`) from kubernetes1014
  - the flink session cluster was able to regain its 24 slots after after 
`1db45eb6-2405-4aa3-bec1-71fcdbbe4f9a` came back (at `2022-02-07T08:07:00`), 
then this POD disappeared again in favor of another one and the service 
successfully restarted.
  - during the whole incident k8s metrics & flink metrics seem to disagree:
- flink says that it lost 4 task managers (1 POD)
- k8s always reports at least 6 PODS 
(`count(container_memory_usage_bytes{namespace="rdf-streaming-updater", 
container="flink-session-cluster-main-taskmanager"})`)
  
  Questions:
  
  - why do flink and k8s metrics disagree (active PODs vs number of task 
manager)?
  - why a new POD was not created after kubernetes1014 went down (making 
`1db45eb6-2405-4aa3-bec1-71fcdbbe4f9a` unavailable to the deployment)?
  
  What could we have done better:
  
  - we could have route wdqs traffic to codfw during the outage and avoid 
throttling edits

TASK DETAIL
  https://phabricator.wikimedia.org/T301147

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T289770: Add hints in response headers for 404 responses in Special:EntityData

2022-02-04 Thread dcausse
dcausse closed this task as "Declined".
dcausse added a comment.


  Thanks for the investigation on this! We don't plan to pursue this route 
given the added complexity and it's not clear if the benefit is worth the 
effort, esp. over tuning retries on 404 and the work on reconciliation. I'm 
tentatively declining and will reopen if we think it's worth reconsidering 
again.

TASK DETAIL
  https://phabricator.wikimedia.org/T289770

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael, dcausse
Cc: Michael, Lydia_Pintscher, Lucas_Werkmeister_WMDE, dcausse, Zbyszko, 
Addshore, Aklapper, 786, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, 
maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T299290: Unexpected behavior in federated queries with LinguaLibre in WDQS

2022-02-01 Thread dcausse
dcausse removed projects: Discovery-Search (Current work), Wikidata, 
Wikidata-Query-Service.
dcausse added a comment.


  Tried to debug this a bit and I believe the problem is on the lingualibre 
side. I suspect a weird bug happening because of the query length.
  Query that passes: https://people.wikimedia.org/~dcausse/T299290-ok.sparql
  Query that fails: https://people.wikimedia.org/~dcausse/T299290-bad.sparql
  Difference is just one empty space.
  Command to run this manually is: `curl -X POST -H"Accept: 
application/sparql-results+xml" --data-urlencode query@T299290-bad.sparql 
-data-urlencode "queryId=de249fca-8362-11ec-a8a3-0242ac120002"  
https://lingualibre.org/sparql`
  
  Happy to help if LinguaLibre maintainers can obtain some server logs.
  Untagging WDQS as there's not much we can do on our side.

TASK DETAIL
  https://phabricator.wikimedia.org/T299290

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Rdrg109, Eihel, Poslovitch, Pamputt, Base, Ltrlg, Invadibot, 
MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T300240: Missing Wikidata RDF (ttl and nt) dumps for 20220117

2022-02-01 Thread dcausse
dcausse added a comment.


  @ArielGlenn no problem! :)
  When these dumps fail we're informed couple days after and it might be 
interesting for us to be pro-active about that but not sure we have enough 
knowledge of the dump process/infra to be super useful in case of failures, if 
you feel it's accessible to a neophyte like me please feel free to add me :)

TASK DETAIL
  https://phabricator.wikimedia.org/T300240

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: ArielGlenn, Aklapper, JAllemandou, AKhatun_WMF, dcausse, Invadibot, 
maantietaja, jannee_e, Akuckartz, holger.knust, Nandana, Lahi, Gq86, 
GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, gnosygnu, Wikidata-bugs, aude, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T300240: Missing Wikidata RDF (ttl and nt) dumps for 20220117

2022-02-01 Thread dcausse
dcausse closed this task as "Invalid".
dcausse added a comment.


  Thanks for checking!

TASK DETAIL
  https://phabricator.wikimedia.org/T300240

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: ArielGlenn, Aklapper, JAllemandou, AKhatun_WMF, dcausse, Invadibot, 
maantietaja, jannee_e, Akuckartz, holger.knust, Nandana, Lahi, Gq86, 
GoranSMilovanovic, Lunewa, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, gnosygnu, Wikidata-bugs, aude, Addshore, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T300240: Missing Wikidata RDF (ttl and nt) dumps for 20220117

2022-01-28 Thread dcausse
dcausse renamed this task from "TaskInstance: 
import_wikidata_ttl.wait_for_all_ttl_dump 2022-01-14T03:00:00+00:00" to 
"Missing Wikidata RDF (ttl and nt) dumps for 20220117".
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T300240

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, JAllemandou, AKhatun_WMF, dcausse, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T300240: TaskInstance: import_wikidata_ttl.wait_for_all_ttl_dump 2022-01-14T03:00:00+00:00

2022-01-28 Thread dcausse
dcausse added a comment.


  RDF dumps seem absent from 
https://dumps.wikimedia.org/wikidatawiki/entities/20220117/ but seems to be 
there again on 20220124. Everything is OK now so I suspect a transient failure. 
Pinging #dumps-generation 
<https://phabricator.wikimedia.org/tag/dumps-generation/> just in case there's 
something to investigate.

TASK DETAIL
  https://phabricator.wikimedia.org/T300240

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, JAllemandou, AKhatun_WMF, dcausse, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

2022-01-27 Thread dcausse
dcausse closed this task as "Resolved".
dcausse added a comment.


  Thanks for the report, blazegraph died I restarted it, should be available 
again now.

TASK DETAIL
  https://phabricator.wikimedia.org/T297454

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcaro, dcausse
Cc: dcausse, Theklan, Marsupium, Vojtech.dostal, Base, RhinosF1, Majavah, 
aborrero, GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD, 
Alicia_Fagerving_WMSE, EBernhardson, Aklapper, Jarekt, Invadibot, MPhamWMF, 
maantietaja, CBogen, Nintendofan885, Akuckartz, Nandana, JKSTNK, Namenlos314, 
Lahi, Gq86, E1presidente, Ramsey-WMF, Cparle, Anoop, SandraF_WMF, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Susannaanas, Jane023, jkroll, Wikidata-bugs, Jdouglas, 
matthiasmullie, aude, Tobias1984, Manybubbles, Ricordisamoa, Wesalius, 
Lydia_Pintscher, Raymond, Steinsplitter, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T300240: TaskInstance: import_wikidata_ttl.wait_for_all_ttl_dump 2022-01-14T03:00:00+00:00

2022-01-27 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  As reported by airflow, a sensor is timing out on:
  
[2022-01-27 04:41:24,843] {hdfs_cli.py:71} INFO - Checking marker at 
hdfs://analytics-hadoop/wmf/data/raw/wikidata/dumps/all_ttl/20220117/_IMPORTED
  
  At the time of the error HDFS only reports this content:
  
hdfs dfs -ls hdfs://analytics-hadoop/wmf/data/raw/wikidata/dumps/all_ttl/
drwxr-x---   - analytics analytics-privatedata-users  0 2022-01-27 
01:30 hdfs://analytics-hadoop/wmf/data/raw/wikidata/dumps/all_ttl/20211213
drwxr-x---   - analytics analytics-privatedata-users  0 2022-01-27 
01:30 hdfs://analytics-hadoop/wmf/data/raw/wikidata/dumps/all_ttl/20211220
drwxr-x---   - analytics analytics-privatedata-users  0 2022-01-27 
01:30 hdfs://analytics-hadoop/wmf/data/raw/wikidata/dumps/all_ttl/20211227
drwxr-x---   - analytics analytics-privatedata-users  0 2022-01-27 
01:30 hdfs://analytics-hadoop/wmf/data/raw/wikidata/dumps/all_ttl/20220103
drwxr-x---   - analytics analytics-privatedata-users  0 2022-01-27 
01:30 hdfs://analytics-hadoop/wmf/data/raw/wikidata/dumps/all_ttl/20220110

TASK DETAIL
  https://phabricator.wikimedia.org/T300240

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, JAllemandou, AKhatun_WMF, dcausse, MPhamWMF, CBogen, Namenlos314, 
Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

2022-01-24 Thread dcausse
dcausse added a comment.


  Sorry for the confusion that the rename I did of this task caused.
  Just to bring clarity on my reasoning as a maintainer of the wikidata query 
service stack as to why being specific on TDB2 might be helpful:
  
  - Some components of Jena are already being used (i.e. the sparql parser for 
query analysis)
  - Jena has been considered in 2015 but declined ref: T90112 
<https://phabricator.wikimedia.org/T90112> (sadly no reasons were given)
  
  This task is I think about evaluating Jena and its storage component as a 
storage/query engine for Wikidata Query Service but it does not mean that all 
of what Jena offers will be discarded if this task is declined.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Osmasuominen, dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, 
Andrawaag, Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, 
So9q, Salgo60, WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, 
Tagishsimon, Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, 
Justin0x2004, Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena TDB2

2022-01-21 Thread dcausse
dcausse renamed this task from "Evaluate Apache Jena" to "Evaluate Apache Jena 
TDB2".
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, Andrawaag, Addshore, 
Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, So9q, Salgo60, 
WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, 
Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, 
Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T299290: Unexpected behavior in federated queries in WDQS

2022-01-17 Thread dcausse
dcausse added a comment.


  WDQS receives `Status Code=502, Status Line=Bad Gateway, Response=` 
from lingualibre servers. I'm not totally sure to understand why it's failing 
esp. why Shopox is generating a query that is accepted there and why it may 
sometimes succeed from wdqs when varying the query.
  
  Few simpler examples that bug me:
  
SELECT * { 
  SERVICE <https://lingualibre.org/sparql> {
select ?item {
  ?item <https://lingualibre.org/prop/direct/P2> 
<https://lingualibre.org/entity/Q5> .
}
  }
}
  
  is NOT OK (OK via shophox)
  
SELECT * { 
  SERVICE <https://lingualibre.org/sparql> {
  ?item <https://lingualibre.org/prop/direct/P2> 
<https://lingualibre.org/entity/Q5> .
  }
}
  
  is OK

TASK DETAIL
  https://phabricator.wikimedia.org/T299290

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Rdrg109, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 
Eihel, Nandana, Namenlos314, Poslovitch, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Pamputt, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, Base, 
aude, Tobias1984, Manybubbles, Mbch331, Ltrlg
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T281468: Automatic SI unit conversion not working on Commons SPARQL engine

2022-01-11 Thread dcausse
dcausse assigned this task to Ladsgroup.
dcausse moved this task from Ready for Development to Needs Reporting on the 
Discovery-Search (Current work) board.
dcausse added a comment.


  I think it's working properly now

TASK DETAIL
  https://phabricator.wikimedia.org/T281468

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup, dcausse
Cc: dcausse, Gehel, Ladsgroup, WMDE-leszek, Addshore, Aklapper, CBogen, 
Lucas_Werkmeister_WMDE, Multichill, Invadibot, GFontenelle_WMF, MPhamWMF, 
maantietaja, Y.ssk, FRomeo_WMF, Muchiri124, Hazizibinmahdi, Nintendofan885, 
Akuckartz, Nandana, JKSTNK, Namenlos314, Lahi, Gq86, E1presidente, Ramsey-WMF, 
Cparle, Anoop, SandraF_WMF, GoranSMilovanovic, QZanden, EBjune, Tramullas, 
Acer, merbst, LawExplorer, Salgo60, Silverfish, Poyekhali, _jensen, rosalieper, 
4nn1l2, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Susannaanas, Ixocactus, 
Wong128hk, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, 
aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, 
Lydia_Pintscher, Raymond, Steinsplitter, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T281468: Automatic SI unit conversion not working on Commons SPARQL engine

2022-01-11 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T281468

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Gehel, Ladsgroup, WMDE-leszek, Addshore, Aklapper, CBogen, 
Lucas_Werkmeister_WMDE, Multichill, Invadibot, GFontenelle_WMF, MPhamWMF, 
maantietaja, Y.ssk, FRomeo_WMF, Muchiri124, Hazizibinmahdi, Nintendofan885, 
Akuckartz, Nandana, JKSTNK, Namenlos314, Lahi, Gq86, E1presidente, Ramsey-WMF, 
Cparle, Anoop, SandraF_WMF, GoranSMilovanovic, QZanden, EBjune, Tramullas, 
Acer, merbst, LawExplorer, Salgo60, Silverfish, Poyekhali, _jensen, rosalieper, 
4nn1l2, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Susannaanas, Ixocactus, 
Wong128hk, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, 
aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, 
Lydia_Pintscher, Raymond, Steinsplitter, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T262265: Provide real-time updates for WCQS

2022-01-05 Thread dcausse
dcausse added a subtask: T298622: Adapt EntityRevisionMapGenerator for wcqs.

TASK DETAIL
  https://phabricator.wikimedia.org/T262265

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson, dcausse
Cc: Back_ache, So9q, Salgo60, Gehel, Aklapper, Zbyszko, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T298622: Adapt EntityRevisionMapGenerator for wcqs

2022-01-05 Thread dcausse
dcausse added a parent task: T262265: Provide real-time updates for WCQS.

TASK DETAIL
  https://phabricator.wikimedia.org/T298622

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T298622: Adapt EntityRevisionMapGenerator for wcqs

2022-01-05 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  This spark job is required for building the initial state of the streaming 
updater, it must be adapted for commons.
  
  AC:
  
  - add new arguments to build the proper UriScheme for commons
  - adapt the import_commons_ttl dag to run it after every import

TASK DETAIL
  https://phabricator.wikimedia.org/T298622

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T240334: Evaluate adding all/some textual properties to the text field

2022-01-03 Thread dcausse
dcausse added a comment.


  Forwarding a suggestion made on 
https://www.wikidata.org/wiki/Wikidata:Report_a_technical_problem/WDQS_and_Search:
  
  > It would be interesting to be able to search for street address 
(P6375)-values, e.g. Special:Search/Getreidegasse Salzburg should find 
Q37970995. --- Jura 19:11, 28 December 2021 (UTC)

TASK DETAIL
  https://phabricator.wikimedia.org/T240334

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Lea_Lacroix_WMDE, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
Wilmanbeno, CBogen, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, 
jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T297870: WDQS Streaming Updater fails with Timeout expired after 60000milliseconds while awaiting InitProducerId

2021-12-16 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  This error causes the pipeline to restart and might trigger the latency alert 
//WdqsStreamingUpdaterFlinkProcessingLatencyIsHigh//.
  
  It was seen on the pipeline running in codfw right after one kafka node was 
removed from the cluster.
  It was not a single instance of the error and it occurred several times 
after, timeline:
  
  - 2021-12-15T16:12 kafka-main2003 is removed from the cluster
  - 2021-12-15T16:17 flink fails
  - 2021-12-15T16:19 flink fails
  - 2021-12-15T16:29 flink fails
  - 2021-12-15T16:37 flink fails
  - 2021-12-15T16:42 flink fails
  - 2021-12-16T10:46 flink fails
  
  The pipeline restarting after a kafka broker is removed is something we 
should expect but the subsequent failures seem to suggest that this setup flink 
+ kafka-main minus one broker is less stable than usual.
  
  Flink is properly resuming without user-facing issues, it's noticeable only 
because the WdqsStreamingUpdaterFlinkProcessingLatencyIsHigh is being triggered.
  
  The flink error stack is:
  
org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete 
snapshot 216620 for operator RDFPatchChunkOperation -> 
MeasureEventProcessingLatencyOperation -> Sink: 
codfw.rdf-streaming-updater.mutation:0 (1/1)#1. Failure reason: Checkpoint was 
declined.
at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:241)
at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:162)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:371)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointStreamOperator(SubtaskCheckpointCoordinatorImpl.java:685)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.buildOperatorSnapshotFutures(SubtaskCheckpointCoordinatorImpl.java:606)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:571)
at 
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:298)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$9(StreamTask.java:1003)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:993)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:951)
at 
org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:115)
at 
org.apache.flink.streaming.runtime.io.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:156)
at 
org.apache.flink.streaming.runtime.io.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:178)
at 
org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:155)
at 
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:179)
at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:395)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:609)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired 
after 6milliseconds while awaiting InitProducerId

error_type
org.apache.flink.runtime.checkpoint.CheckpointException

TASK DETAIL
  https://phabricator.wikimedia.org/T297870

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: BTullis, elukey, dcausse, Aklapper, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294076: Blazegraph and MariaDB contain different sitelinks at Wikidata

2021-11-29 Thread dcausse
dcausse merged a task: T295941: WDQS Data drift.
dcausse added subscribers: William_Avery, dcausse.

TASK DETAIL
  https://phabricator.wikimedia.org/T294076

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, William_Avery, RShigapov, Aklapper, Invadibot, MPhamWMF, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T295941: WDQS Data drift

2021-11-29 Thread dcausse
dcausse closed this task as a duplicate of T294076: Blazegraph and MariaDB 
contain different sitelinks at Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T295941

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, William_Avery, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T295941: WDQS Data drift

2021-11-29 Thread dcausse
dcausse added a comment.


  @William_Avery thanks for the report this is very helpful.
  
  I found almost all listed items in the "fetch failures" data set and these 
will be corrected once we have T279541 
<https://phabricator.wikimedia.org/T279541> in place.
  Remaining ones are:
  
  - Q3130156: still has the statement //Hemiergis quadrilineatum// using P31 
<https://phabricator.wikimedia.org/P31>
  - Q5705943: is no longer returned by your query
  - Q14828500: still has the statement //Acalolepta pseudotincturata// using 
P31 <https://phabricator.wikimedia.org/P31>
  - Q21191097: still has the statement //Forpus spengelli// using P31 
<https://phabricator.wikimedia.org/P31>
  
  I'm closing as a duplicate of T294076 
<https://phabricator.wikimedia.org/T294076> because the root cause is identical 
and will be solved by T279541 <https://phabricator.wikimedia.org/T279541>.

TASK DETAIL
  https://phabricator.wikimedia.org/T295941

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, William_Avery, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T280485: Additional capacity on the k8s Flink cluster for WCQS updater

2021-11-17 Thread dcausse
dcausse added a comment.


  small precision:
  If we reuse the same cluster (same k8s namescape):
  
  - it's 3 more pods at 2.1G ram, cpu: 1000m each
  
  If we reuse a separate cluster (new k8s namescape):
  
  - add a pod at 1.6G, cpu: 500m to the 3 pods mentioned above

TASK DETAIL
  https://phabricator.wikimedia.org/T280485

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel, dcausse
Cc: dcausse, akosiaris, Zbyszko, Aklapper, RKemper, Gehel, MPhamWMF, wkandek, 
JMeybohm, CBogen, Namenlos314, jijiki, Gq86, Lucas_Werkmeister_WMDE, EBjune, 
merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293063: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes

2021-11-09 Thread dcausse
dcausse added a comment.


  In T293063#7491903 <https://phabricator.wikimedia.org/T293063#7491903>, 
@JMeybohm wrote:
  
  > @dcausse IIRC we said that "something in the areas of hours" would be 
considered a "short maintenance" and thus would not need any additional actions 
to be carried out, right?
  
  We are targeting a SLO with an update lag below 10minutes for 99% of the 
time, we are still learning what is the operational cost of this and are happy 
to discuss/re-adjust all this depending on your constraints.
  
  > As part of T251305 <https://phabricator.wikimedia.org/T251305> we will 
re-create the helm release of flink in both datacenters (one after the other 
ofc.) and that would mean flink will be down for a couple of minutes. If my 
memory and understanding is still intact, the checkpoint/tombstone metadata is 
not part of the helm release itself (it's in those flink managed configmaps). 
So it should survive purging and recreating the helm release.
  
  Yes if the configmaps are kept flink will just autorestart on its own, 
regarding lag I'm not worried as already flink restarts on its own from time to 
time without affecting the 10min lag SLO.
  
  > @Jelto has alredy done that for the staging flink release. If you have the 
chance it would be nice if you could double check that is still working as 
expected.
  
  Checking the logs I see 2 restarts in the last 7 days and both restarts 
properly restored the job:
  
Nov 3, 2021 @ 15:44:33.739  syslog  kubestage1002   Restoring job 
095b671d83457ebf4c59166fda7a7055 from Checkpoint 106609 @ 1635954210959 for 
095b671d83457ebf4c59166fda7a7055 located at 
swift://rdf-streaming-updater-staging.thanos-swift/wikidata/checkpoints/095b671d83457ebf4c59166fda7a7055/chk-106609.

Nov 4, 2021 @ 13:36:35.097  syslog  kubestage1002   Restoring job 
095b671d83457ebf4c59166fda7a7055 from Checkpoint 109216 @ 1636032918483 for 
095b671d83457ebf4c59166fda7a7055 located at 
swift://rdf-streaming-updater-staging.thanos-swift/wikidata/checkpoints/095b671d83457ebf4c59166fda7a7055/chk-109216.
  
  So, if one of these restarts corresponds to the helm 3 upgrade then I can 
confirm that it will work properly the production clusters.
  
  > Besides that I tried to understand what would be needed to do for a "longer 
downtime" of k8s and it's not exactly clear to me. Could we have a dedicated 
section for that on whe wikitech page? IIRC that also needed a change to WQDS 
itself.
  
  Certainly, this task is all about clarifying all this.

TASK DETAIL
  https://phabricator.wikimedia.org/T293063

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: JMeybohm, Jelto, Aklapper, jijiki, dcausse, Invadibot, MPhamWMF, 
GeminiAgaloos, maantietaja, wkandek, CBogen, Akuckartz, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T279541: Add a reconciliation strategy to the wdqs streaming updater

2021-11-08 Thread dcausse
dcausse claimed this task.
dcausse moved this task from Incoming to In Progress on the Discovery-Search 
(Current work) board.

TASK DETAIL
  https://phabricator.wikimedia.org/T279541

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: RShigapov, dcausse, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293195: Add MCR slot information to revision-create events

2021-10-27 Thread dcausse
dcausse added a comment.


  In T293195#7459268 <https://phabricator.wikimedia.org/T293195#7459268>, 
@Ottomata wrote:
  
  > I was about to merge that today but then thought that your suggestion to 
ensure that properties validate with the additionalProperties stuff would be 
good to add first.  So you could implement that :D :D
  
  Sure! I took a look and will amend your patch :)

TASK DETAIL
  https://phabricator.wikimedia.org/T293195

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: odimitrijevic, Cparle, JAllemandou, Milimetric, Aklapper, Ottomata, 
Pchelolo, dcausse, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, 
MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, 4748kitoko, Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, 
Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T293195: Add MCR slot information to revision-create events

2021-10-26 Thread dcausse
dcausse added a comment.


  This is blocked on 
https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/629406 which is 
required to support the new pattern (additionalProperties + properties). 
@Ottomata is there anything we could do help unblock the work on your refinery 
patch?

TASK DETAIL
  https://phabricator.wikimedia.org/T293195

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: odimitrijevic, Cparle, JAllemandou, Milimetric, Aklapper, Ottomata, 
Pchelolo, dcausse, EChetty, Suran38, Biggs657, Invadibot, Lalamarie69, 
MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, 4748kitoko, Hook696, Kent7301, holger.knust, joker88john, CucyNoiD, 
Nandana, Namenlos314, Akovalyov, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, 
Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, GWicke, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294076: Blazegraph and MariaDB contain different sitelinks at Wikidata

2021-10-26 Thread dcausse
dcausse moved this task from In Progress to Waiting on the Discovery-Search 
(Current work) board.
dcausse added a comment.


  Thanks for the report this is very helpful.
  In the two updates you mention here were missed by the new updater but both 
of these were properly identified as problematic and will be resolved once we 
have the reconciliation strategy (work tracked in T279541 
<https://phabricator.wikimedia.org/T279541>)
  
  For the record here are the notes regarding these two missed updates:
  
  | edit time| item | wikibase truth | old updater wdqs1010 | wdqs 
eqiad wdqs1009 | wdqs codfw wdqs2008 | in revision-create topic | in mutation 
topic | in fetch-failure   |
  | 2021-10-11T17:11:00‎ | Q6766777| 1510811138  | 1510811138   
 | 1392684585   | 1510811138   
| yes  | codfw only| none (only in raw for eqiad 
T294361 <https://phabricator.wikimedia.org/T294361>) |
  | 2021-10-17T6:19:36   | Q19929406| 1512982605  | deleted 
 | 1512453868   | 1512453868   | yes
  | no| eqiad|
  |
  
  I'm moving this to waiting while T279541 
<https://phabricator.wikimedia.org/T279541> is being worked out so that we have 
a place future inconsistencies.

TASK DETAIL
  https://phabricator.wikimedia.org/T294076

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: RShigapov, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T279541: Add a reconciliation strategy to the wdqs streaming updater

2021-10-26 Thread dcausse
dcausse added a subtask: T294361: Events missing from 
event.rdf_streaming_updater_fetch_failure but present in 
/wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure.

TASK DETAIL
  https://phabricator.wikimedia.org/T279541

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure

2021-10-26 Thread dcausse
dcausse added a parent task: T279541: Add a reconciliation strategy to the wdqs 
streaming updater.

TASK DETAIL
  https://phabricator.wikimedia.org/T294361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, EChetty, MPhamWMF, CBogen, 4748kitoko, Namenlos314, 
Akovalyov, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, 
JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294361: Events missing from event.rdf_streaming_updater_fetch_failure but present in /wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure

2021-10-26 Thread dcausse
dcausse created this task.
dcausse added projects: Analytics, Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  While investigating missed updates on the WDQS streaming updater I looked at 
our flink side-outputs that record all the failures happening while fetching 
the data out of MW.
  
  For instance I found that we missed the revision 1510811138 for the wikidata 
entity Q6766777 (edit that occurred at `2021-10-11T17:11:00Z`).
  The fetch for this entity had failed because of a 404 hitting MW (due to the 
race between the events & mysql replication). This type of failure is recorded 
in the topics [eqiad|codfw].rdf-streaming-updater.fetch-failure.
  
  Looking at the raw data I can find this record:
  
hdfs dfs -text 
/wmf/data/raw/event/eqiad.rdf-streaming-updater.fetch-failure/year=2021/month=10/day=11/hour=17/part.task_event_default_1633975518391_41_0.txt.gz
 | grep Q6766777 | jq .
  
{
  "meta": {
"domain": "www.wikidata.org",
"dt": "2021-10-11T17:11:00.853232Z",
"stream": "rdf-streaming-updater.fetch-failure"
  },
  "item": "Q6766777",
  "original_ingestion_dt": "2021-10-11T17:11:00.732866Z",
  "revision_id": 1510811138,
  "original_event_info": {
"dt": "2021-10-11T17:11:00Z",
"$schema": "/mediawiki/revision/create/1.1.0",
"meta": {
  "id": "b02f7b80-05d0-4060-9808-bcaef6a2304e",
  "dt": "2021-10-11T17:11:00Z",
  "stream": "mediawiki.revision-create",
  "request_id": "d1297274-7e9e-4de9-a9f4-63d86d3b566a",
  "domain": "www.wikidata.org"
}
  },
  "op_type": "diff",
  "from_revision_id": 1392684585,
  "exception_type": 
"org.wikidata.query.rdf.tool.wikibase.WikibaseEntityFetchException",
  "exception_msg": "Cannot fetch entity at 
https://www.wikidata.org/wiki/Special:EntityData/Q6766777.ttl?flavor=dump=1510811138:
 ENTITY_NOT_FOUND",
  "fetch_error_type": "ENTITY_NOT_FOUND",
  "dt": "2021-10-11T17:11:00.853125Z",
  "$schema": "/rdf_streaming_updater/fetch_failure/1.0.0"
}
  
  But looking at the refined table I can't find this same record:
  
select item, meta.dt from event.rdf_streaming_updater_fetch_failure where 
year = 2021 and month = 10 and day = 11 and item = "Q6766777";
  
  finds nothing.

TASK DETAIL
  https://phabricator.wikimedia.org/T294361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, EChetty, MPhamWMF, CBogen, 4748kitoko, Namenlos314, 
Akovalyov, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, 
JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294076: Blazegraph and MariaDB contain different sitelinks at Wikidata

2021-10-26 Thread dcausse
dcausse claimed this task.
dcausse moved this task from Ready for Development to In Progress on the 
Discovery-Search (Current work) board.

TASK DETAIL
  https://phabricator.wikimedia.org/T294076

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: RShigapov, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

2021-10-22 Thread dcausse
dcausse added a subscriber: Ottomata.
dcausse added a project: EventStreams.
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Ottomata, Aklapper, dcausse, MPhamWMF, RBrounley_WMF, CBogen, Namenlos314, 
Gq86, Xinbenlv, Vacio, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, 
Nirmos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Krenair
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

2021-10-22 Thread dcausse
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

2021-10-22 Thread dcausse
dcausse created this task.
dcausse added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  As a consumer of the wikidata content I want to be able to have access the 
same RDF data the WMF WDQS servers  use to perform their live updates so that I 
can keep my own replica of the wikidata query service (or another RDF store) up 
to date more easily.
  
  A solution might be to use the EventStreams 
<https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams> service.
  
  Note on the stream:
  It was decided to go fully active/active for the flink application powering 
the WDQS updater. Which means the complete stream of changes is available in 
both topics:
  
  - eqiad.rdf-streaming-updater.mutation
  - codfw.rdf-streaming-updater.mutation
  
  It is sligthly different to what we currently see in our topic topology where 
if you want to have a complete view of the data you need to consume both 
eqiad.topic and codfw.topic. Here you must consume only one.
  
  AC:
  
  - RDF data is exposed through EventStreams
  - A java client is offered for third parties to use with store comptatible 
with the SPARQL 1.1 Update operations 
<https://www.w3.org/TR/2013/REC-sparql11-protocol-20130321/#update-operation>

TASK DETAIL
  https://phabricator.wikimedia.org/T294133

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2021-10-22 Thread dcausse
dcausse closed subtask T266321: Determine flink metrics configuration and 
backend when running from k8s as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T244590

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Mohammed_Sadat_WMDE, So9q, Lydia_Pintscher, VladimirAlexiev, karapayneWMDE, 
MPhamWMF, Daniel_Mietchen, Thadguidry, tfmorris, revi, Ladsgroup, Multichill, 
darthmon_wmde, Iamamz3, Smalyshev, Ottomata, JAllemandou, Aklapper, Zbyszko, 
Gehel, dcausse, Suran38, Invadibot, maantietaja, Peteosx1x, NavinRizwi, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Dinoguy1000, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T266321: Determine flink metrics configuration and backend when running from k8s

2021-10-22 Thread dcausse
dcausse closed this task as "Resolved".
dcausse claimed this task.
dcausse added a comment.


  updater specific metrics are available here: 
https://grafana-rw.wikimedia.org/d/fdU5Zx-Mk/wdqs-streaming-updater?orgId=1
  flink specific metrics are available here: 
https://grafana-rw.wikimedia.org/d/gCFgfpG7k/flink-session-cluster?orgId=1

TASK DETAIL
  https://phabricator.wikimedia.org/T266321

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


  1   2   3   4   5   6   7   8   9   10   >