[Wikidata-bugs] [Maniphest] T292621: INVESTIGATION: Fix stats for Wikidata dump downloads dashboard

2022-06-15 Thread Manuel 
Manuel closed this task as "Resolved".
Manuel moved this task from Prioritized Backlog to Our work done on the 
Wikidata-Campsite (Team A Hearth 🏰🔥) board.
Manuel claimed this task.
Manuel added a comment.


  Yes, this helped a lot, thank you Lucas! \o/
  
  So basically, the data is valid. The shape of the curve is by nature highly 
influenced by a small number of agents that make a lot of successful requests. 
"Go-http-client/1.1" e.g. seems to be related to Google.
  
  The high number of 206s seems to have resolved itself. So it seems that there 
is nothing fundamental to worry about.

TASK DETAIL
  https://phabricator.wikimedia.org/T292621

WORKBOARD
  https://phabricator.wikimedia.org/project/board/5612/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Michael, Manuel, Lucas_Werkmeister_WMDE, Aklapper, Lydia_Pintscher, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292621: INVESTIGATION: Fix stats for Wikidata dump downloads dashboard

2022-06-15 Thread ItamarWMDE 
ItamarWMDE moved this task from Tech backlog to Special:NewLexeme revival - 
sprint 10 on the Special:NewLexeme revival board.
ItamarWMDE edited projects, added Special:NewLexeme revival (Special:NewLexeme 
revival - sprint 10); removed Special:NewLexeme revival.

TASK DETAIL
  https://phabricator.wikimedia.org/T292621

WORKBOARD
  https://phabricator.wikimedia.org/project/board/5674/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ItamarWMDE
Cc: Michael, Manuel, Lucas_Werkmeister_WMDE, Aklapper, Lydia_Pintscher, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292621: INVESTIGATION: Fix stats for Wikidata dump downloads dashboard

2022-06-15 Thread Manuel 
Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T292621

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Michael, Manuel, Lucas_Werkmeister_WMDE, Aklapper, Lydia_Pintscher, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292621: INVESTIGATION: Fix stats for Wikidata dump downloads dashboard

2022-06-14 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  > The recent decline in downloads (after 17.3.2022) and the changed shape of 
the line are looking a bit suspicious. Something seems to have changed, 
probably in the way we are measuring this. It is important to understand this 
well enough to interpret the curve right. Could you please add a description to 
the board that documents what has happened?
  
  The pattern seems to be real, at least when looking at successful requests 
(status 200):
  
lucaswerkmeister-wmde@stat1007:~$ zgrep -F /wikidatawiki/ 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-202203*.gz | grep -E 
-e '(latest|wikidata-[0-9]{8})-all\.json\.(gz|bz2)' | grep -F ' 200 ' | cut -d: 
-f1 | uniq -c
   1166 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220301.gz
   1281 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220302.gz
   1236 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220303.gz
   1022 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220304.gz
   1317 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220305.gz
   1164 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220306.gz
   1095 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220307.gz
   1200 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220308.gz
   1171 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220309.gz
   1059 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220310.gz
   1044 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220311.gz
   1184 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220312.gz
   1107 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220313.gz
   1021 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220314.gz
   1242 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220315.gz
   1189 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220316.gz
987 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220317.gz
961 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220318.gz
550 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220319.gz
126 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220320.gz
138 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220321.gz
285 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220322.gz
203 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220323.gz
227 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220324.gz
342 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220325.gz
231 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220326.gz
129 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220327.gz
196 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220328.gz
298 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220329.gz
269 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220330.gz
308 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220331.gz
  
  (Note that the 206 responses (partial content) are much more erratic:)
  
lucaswerkmeister-wmde@stat1007:~$ zgrep -F /wikidatawiki/ 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-202203*.gz | grep -E 
-e '(latest|wikidata-[0-9]{8})-all\.json\.(gz|bz2)' | grep -F ' 206 ' | cut -d: 
-f1 | uniq -c
 60 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220301.gz
120 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220302.gz
419 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220303.gz
321 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220304.gz
105 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220305.gz
120 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220306.gz
   1530 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220307.gz
   1359 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220308.gz
182 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220309.gz
   1273 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220310.gz
400 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220311.gz
352 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220312.gz
 82 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220313.gz
381 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220314.gz
326 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220315.gz
286 
/srv/log/webrequest/archive/dumps.wikimedia.org/access.log-20220316.

[Wikidata-bugs] [Maniphest] T292621: INVESTIGATION: Fix stats for Wikidata dump downloads dashboard

2022-06-14 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  In T292621#8001541 , 
@Manuel wrote:
  
  > In what way are T162346  and 
T218711  related?
  
  I assume T218711: Regular wikidata JSON dump scanning broken on analytics 
machine  is unrelated – I don’t know 
what those scans did, but if they’ve been broken for years then I don’t think 
it can be directly related to this task.
  
  T162346: Include truthy nt dumps in the Wikidata Dump Downloads Grafana 
dashboard  looks like it would still 
be relevant (the truthy dumps still aren’t tracked as far as I can tell), but 
shouldn’t affect the download numbers for the full / incremental dumps.

TASK DETAIL
  https://phabricator.wikimedia.org/T292621

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Michael, Manuel, Lucas_Werkmeister_WMDE, Aklapper, Lydia_Pintscher, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292621: INVESTIGATION: Fix stats for Wikidata dump downloads dashboard

2022-06-14 Thread Manuel 
Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T292621

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Michael, Manuel, Lucas_Werkmeister_WMDE, Aklapper, Lydia_Pintscher, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292621: INVESTIGATION: Fix stats for Wikidata dump downloads dashboard

2022-06-14 Thread karapayneWMDE 
karapayneWMDE renamed this task from "Fix stats for Wikidata dump downloads 
dashboard" to "INVESTIGATION: Fix stats for Wikidata dump downloads dashboard".
karapayneWMDE updated the task description.
karapayneWMDE set the point value for this task to "1".

TASK DETAIL
  https://phabricator.wikimedia.org/T292621

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: karapayneWMDE
Cc: Michael, Manuel, Lucas_Werkmeister_WMDE, Aklapper, Lydia_Pintscher, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org