[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[Addshore]

Addshore closed this task as "Resolved".
Addshore claimed this task.
Restricted Application added a project: User-Addshore.

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
CBogen, Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[WMDE-leszek]

WMDE-leszek added a comment.


  It is not entirely clear to me what is the expected behaviour. Looking at 
https://logstash.wikimedia.org/goto/cb4883b25c11556429eed17ec7e78dcd there seem 
to just be a handful of `{method}: Request to Prometheus API {apiUrl} failed 
with {error}` errors over last 24 hours. They all seem to be related to some 
network problems
  
  > 
WikidataOrg\QueryServiceLag\WikimediaPrometheusQueryServiceLagProvider::getLags:
 Request to Prometheus API 
http://prometheus.svc.eqiad.wmnet/ops/api/v1/query?query=blazegraph_lastupdated 
failed with There was a problem during the HTTP request: 503 Service Unavailable
  
  I suspect we are conveniently going to label these as a temporary hiccup? Is 
there anything else to resolve within this task? @Addshore

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: WMDE-leszek
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
CBogen, Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[CBogen]

CBogen edited projects, added Discovery-Search; removed Discovery-Search 
(Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: CBogen
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
CBogen, Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
CBogen, Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, Namenlos314, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, 
Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[ReleaseTaggerBot]

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.11; 2020-09-29).

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ReleaseTaggerBot
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Iflorez, 
darthmon_wmde, Kent7301, alaa_wmde, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[gerritbot]

gerritbot added a comment.


  Change 630205 **merged** by jenkins-bot:
  [mediawiki/extensions/Wikidata.org@master] Don't log an error on broken 
timestamp for irrelevant clusters
  
  https://gerrit.wikimedia.org/r/630205

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Iflorez, 
darthmon_wmde, Kent7301, alaa_wmde, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[Michael]

Michael added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞).
Michael added a comment.


  Adding it to camp and moving it to review

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Iflorez, 
darthmon_wmde, Kent7301, alaa_wmde, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Jdforrester-WMF, Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[gerritbot]

gerritbot added a comment.


  Change 630205 had a related patch set uploaded (by Michael Große; owner: 
Michael Große):
  [mediawiki/extensions/Wikidata.org@master] Don't log an error on broken 
timestamp for irrelevant clusters
  
  https://gerrit.wikimedia.org/r/630205

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, darthmon_wmde, 
Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, Mbch331, Rxy, Jay8g, 
Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[Pablo-WMDE]

Pablo-WMDE added a comment.


  Copying conversations from chat
  
  @Michael wrote
  
  > This is probably fine because it lets us know that there is a problem that 
is not unexpected but needs fixing.
  > The problem seems to be that this includes wdqs-test servers for which the 
premise "should always provide correct results" does not hold?
  
  @Addshore wrote
  
  > Yes, it looks like wdqs-test is the issue then, and we shouldnt be warning 
for the clusters that we are not actually going to care about

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pablo-WMDE
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[Addshore]

Addshore added a comment.


  Although, the above should have already been patched 
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikidata.org/+/597474
  
  Looks like it needs more investigation!

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[Addshore]

Addshore added a comment.


addshore@deploy1001:~$ curl 
http://prometheus.svc.eqiad.wmnet/ops/api/v1/query?query=blazegraph_lastupdated

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"blazegraph_lastupdated","cluster":"wdqs","instance":"wdqs1004:9193","job":"blazegraph","site":"eqiad"},"value":[1600935337.656,"1600935288"]},{"metric":{"__name__":"blazegraph_lastupdated","cluster":"wdqs","instance":"wdqs1005:9193","job":"blazegraph","site":"eqiad"},"value":[1600935337.656,"1600935310"]},{"metric":{"__name__":"blazegraph_lastupdated","cluster":"wdqs","instance":"wdqs1006:9193","job":"blazegraph","site":"eqiad"},"value":[1600935337.656,"1600935295"]},{"metric":{"__name__":"blazegraph_lastupdated","cluster":"wdqs","instance":"wdqs1007:9193","job":"blazegraph","site":"eqiad"},"value":[1600935337.656,"1600935296"]},{"metric":{"__name__":"blazegraph_lastupdated","cluster":"wdqs-internal","instance":"wdqs1003:9193","job":"blazegraph","site":"eqiad"},"value":[1600935337.656,"1600935287"]},{"metric":{"__name__":"blazegraph_lastupdated","cluster":"wdqs-internal","instance":"wdqs1008:9193","job":"blazegraph","site":"eqiad"},"value":[1600935337.656,"1600935313"]},{"metric":{"__name__":"blazegraph_lastupdated","cluster":"wdqs-test","instance":"wdqs1009:9193","job":"blazegraph","site":"eqiad"},"value":[1600935337.656,"NaN"]},{"metric":{"__name__":"blazegraph_lastupdated","cluster":"wdqs-test","instance":"wdqs1009:9194","job":"blazegraph","site":"eqiad"},"value":[1600935337.656,"NaN"]},{"metric":{"__name__":"blazegraph_lastupdated","cluster":"wdqs-test","instance":"wdqs1010:9193","job":"blazegraph","site":"eqiad"},"value":[1600935337.656,"1600935298"]}]}}
  
  prettified
  
{
"status": "success",
"data":
{
"resultType": "vector",
"result": [
{
"metric":
{
"__name__": "blazegraph_lastupdated",
"cluster": "wdqs",
"instance": "wdqs1004:9193",
"job": "blazegraph",
"site": "eqiad"
},
"value": [1600935337.656, "1600935288"]
},
{
"metric":
{
"__name__": "blazegraph_lastupdated",
"cluster": "wdqs",
"instance": "wdqs1005:9193",
"job": "blazegraph",
"site": "eqiad"
},
"value": [1600935337.656, "1600935310"]
},
{
"metric":
{
"__name__": "blazegraph_lastupdated",
"cluster": "wdqs",
"instance": "wdqs1006:9193",
"job": "blazegraph",
"site": "eqiad"
},
"value": [1600935337.656, "1600935295"]
},
{
"metric":
{
"__name__": "blazegraph_lastupdated",
"cluster": "wdqs",
"instance": "wdqs1007:9193",
"job": "blazegraph",
"site": "eqiad"
},
"value": [1600935337.656, "1600935296"]
},
{
"metric":
{
"__name__": "blazegraph_lastupdated",
"cluster": "wdqs-internal",
"instance": "wdqs1003:9193",
"job": "blazegraph",
"site": "eqiad"
},
"value": [1600935337.656, "1600935287"]
},
{
"metric":
{
"__name__": "blazegraph_lastupdated",
"cluster": "wdqs-internal",
"instance": "wdqs1008:9193",
"job": "blazegraph",
"site": "eqiad"
},
"value": [1600935337.656, "1600935313"]
},
{
"metric":
{
"__name__": "blazegraph_lastupdated",
"cluster": "wdqs-test",
"instance": "wdqs1009:9193",
 

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[Pablo-WMDE]

Pablo-WMDE added subscribers: hoo, WMDE-leszek.
Pablo-WMDE added a comment.


  I have no idea where to even begin to debug this. Maybe one of the 
subscribers has an idea. Triage by someone in the know would also be great.

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pablo-WMDE
Cc: WMDE-leszek, hoo, Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, 
CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[Pablo-WMDE]

Pablo-WMDE added a project: Wikidata.org.

TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pablo-WMDE
Cc: Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, CBogen, 
Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263670: Prometheus API error when requesting Blazegraph lag for WDQS

[Michael]

Michael created this task.
Michael added projects: Wikimedia-production-error, Wikidata, 
Wikidata-Query-Service, Discovery-Search (Current work).
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  Error
  -
  
  `MediaWiki version:`  **`1.36.0-wmf.9`**
  
  name=message

WikidataOrg\QueryServiceLag\WikimediaPrometheusQueryServiceLagProvider::getLags:
 unexpected result from Prometheus API 
http://prometheus.svc.eqiad.wmnet/ops/api/v1/query?query=blazegraph_lastupdated
  
  
  
  name=cli_argv
/srv/mediawiki-staging/multiversion/MWScript.php 
extensions/Wikidata.org/maintenance/updateQueryServiceLag.php --wiki 
wikidatawiki --cluster wdqs --prometheus prometheus.svc.eqiad.wmnet 
--prometheus prometheus.svc.codfw.wmnet
  
  
  
  Impact
  --
  
  - There seems to be a very consistent warning (every 30 seconds) from some 
Prometheus API for requesting some blazegraph/WDQS lag: 
https://logstash.wikimedia.org/goto/9d2f816a8e0a38514b3e8e4d794eddc2
  
  F32361061: Image Pasted at 2020-9-23 18-46.png 

  
  Notes
  -
  
  - It started last Friday (2020-09-18)
  - tagging also #discovery-search-sprint 
 as there seems 
to be some work happening on WDQS/Blazegraph in that context as well
  - I cant see anything out of the ordinary on the WDQS grafana board 
,
 except that there seems to have been a jump in the load of one of the servers 
in the evening of the 17th. But that might be unrelated: F32361085: image.png 


TASK DETAIL
  https://phabricator.wikimedia.org/T263670

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: Aklapper, Addshore, Pablo-WMDE, Lydia_Pintscher, Michael, CBogen, 
Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Jdforrester-WMF, 
Mbch331, Rxy, Jay8g, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs