[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-10-01 Thread mpopov 
mpopov closed this task as a duplicate of T287381: External referrer  WDQS 
metrics stopped updating on 2021-04-25.

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mpopov
Cc: SWakiyama, MPhamWMF, dcausse, mpopov, Zbyszko, Aklapper, Lydia_Pintscher, 
Invadibot, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-10-01 Thread mpopov 
mpopov added a comment.


  Thanks @MPhamWMF!
  
  What Mike and David said is correct. Also, this ticket prompted me to finally 
add the decommission notice to the dashboard (previously it was only on the 
homepage).
  
  In T292152#7391826 , 
@Lydia_Pintscher wrote:
  
  > In the meantime for my talk: Do we know what the current number is?
  
  For 2021-09-30:
  
  | Path  | "Automated" | "User" | Total |
  | - | --- | -- | - |
  | / | 2109| 2290   | 4399  |
  | /bigdata/ldf  | 4   | 55230  | 55234 |
  | /bigdata/namespace/wdq/sparql | 1835762   | 5786966  | 7622728  
   |
  |
  
  Anyone with private data access can easily count 1 day's requests using Hue 
 and this Hive query (slightly modified from 
https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/discovery/golden/+/refs/heads/master/modules/metrics/wdqs/basic_usage):
  
USE wmf;
SELECT
  year, month, day,
  IF(uri_path = '/sparql', '/bigdata/namespace/wdq/sparql', uri_path) AS 
path,
  UPPER(http_status IN('200','304')) as http_success,
  CASE
WHEN (
  agent_type = 'user' AND (
user_agent RLIKE 'https?://'
OR INSTR(user_agent, 'www.') > 0
OR INSTR(user_agent, 'github') > 0
OR LOWER(user_agent) RLIKE 
'([a-z0-9._%-]+@[a-z0-9.-]+\.(com|us|net|org|edu|gov|io|ly|co|uk))'
OR (
  user_agent_map['browser_family'] = 'Other'
  AND user_agent_map['device_family'] = 'Other'
  AND user_agent_map['os_family'] = 'Other'
  )
)
) OR agent_type = 'spider' THEN 'TRUE'
ELSE 'FALSE' END AS is_automata,
  COUNT(*) AS events
FROM wmf.webrequest
WHERE
  webrequest_source = 'text'
  AND year = ${year} AND month = ${month} AND day = ${day}
  AND uri_host = 'query.wikidata.org'
  AND uri_path IN('/', '/bigdata/namespace/wdq/sparql', '/bigdata/ldf', 
'/sparql')
GROUP BY
  year, month, day,
  IF(uri_path = '/sparql', '/bigdata/namespace/wdq/sparql', uri_path),
  UPPER(http_status IN('200','304')),
  CASE
WHEN (
  agent_type = 'user' AND (
user_agent RLIKE 'https?://'
OR INSTR(user_agent, 'www.') > 0
OR INSTR(user_agent, 'github') > 0
OR LOWER(user_agent) RLIKE 
'([a-z0-9._%-]+@[a-z0-9.-]+\.(com|us|net|org|edu|gov|io|ly|co|uk))'
OR (
  user_agent_map['browser_family'] = 'Other'
  AND user_agent_map['device_family'] = 'Other'
  AND user_agent_map['os_family'] = 'Other'
  )
)
) OR agent_type = 'spider' THEN 'TRUE'
ELSE 'FALSE' END;
  
  **I would NOT recommend querying an entire month with 1 query** since it uses 
webrequest data which **should be queried 1 day at a time at most**. Also, the 
query uses non-standard "automata" determination. At the time (those years ago) 
I thought it was clever, but these days I would not use those rules and if I 
had infinite time I would switch to 
https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/BotDetection

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: mpopov
Cc: SWakiyama, MPhamWMF, dcausse, mpopov, Zbyszko, Aklapper, Lydia_Pintscher, 
Invadibot, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-09-30 Thread MPhamWMF 
MPhamWMF added a subscriber: SWakiyama.
MPhamWMF added a comment.


  @dcausse @Lydia_Pintscher  @SWakiyama, it looks like the number of triples 
continues here starting June 2021: 
https://grafana.wikimedia.org/d/00489/wikidata-query-service?viewPanel=7=1=1m=now-6M=now
  David mentioned that data stopped being officially updated in Aug 2021, but 
filtering to the last 30 days seems to show that we're still tracking growth: 
https://grafana.wikimedia.org/d/00489/wikidata-query-service?viewPanel=7=1=1m=now-30d=now
  
  Lydia, Aisha's basic analysis might also be helpful for some numbers: 
https://wikitech.wikimedia.org/wiki/User:AKhatun/Wikidata_Basic_Analysis

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MPhamWMF
Cc: SWakiyama, MPhamWMF, dcausse, mpopov, Zbyszko, Aklapper, Lydia_Pintscher, 
Invadibot, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-09-30 Thread Lydia_Pintscher 
Lydia_Pintscher added a comment.


  In the meantime for my talk: Do we know what the current number is?

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: MPhamWMF, dcausse, mpopov, Zbyszko, Aklapper, Lydia_Pintscher, Invadibot, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-09-30 Thread Lydia_Pintscher 
Lydia_Pintscher added a comment.


  Ah I think it would be very unfortunate if this data isn't available publicly 
anymore. It's a pretty key thing for all of us and the community to be keeping 
an eye on imho.

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: MPhamWMF, dcausse, mpopov, Zbyszko, Aklapper, Lydia_Pintscher, Invadibot, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-09-30 Thread dcausse 
dcausse added a comment.


  @MPhamWMF see T227782 , the data 
stopped to be officially updated on Aug 2021 (even though something in this 
data pipeline seemed to have broke earlier around April).

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: MPhamWMF, dcausse, mpopov, Zbyszko, Aklapper, Lydia_Pintscher, Invadibot, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-09-30 Thread MPhamWMF 
MPhamWMF added subscribers: Zbyszko, mpopov, dcausse, MPhamWMF.
MPhamWMF added a comment.


  I remember that @mpopov moved that graph to 
https://www.mediawiki.org/wiki/User:MPopov_(WMF)/Wikimania_2021_Hackathon in 
anticipation of decommissioning the link you included in the ticket. But I 
don't think it includes any new data since April 2021.
  
  @dcausse @Zbyszko , do I remember correctly that there is some unknown error 
that has caused us to not have data since April 2021?

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MPhamWMF
Cc: MPhamWMF, dcausse, mpopov, Zbyszko, Aklapper, Lydia_Pintscher, Invadibot, 
maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-09-30 Thread Maintenance_bot 
Maintenance_bot added a project: Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: Aklapper, Lydia_Pintscher, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-09-30 Thread Lydia_Pintscher 
Lydia_Pintscher added a comment.


  It'd also help me to get a current number for daily queries already for my 
WikidataCon talk.

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Aklapper, Lydia_Pintscher, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T292152: dashboard with daily query service usage not updating

2021-09-30 Thread Lydia_Pintscher 
Lydia_Pintscher created this task.
Lydia_Pintscher added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  https://discovery.wmflabs.org/wdqs/#wdqs_usage has no new data since April 
2021 currently. It'd be good to get this to update again.

TASK DETAIL
  https://phabricator.wikimedia.org/T292152

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Aklapper, Lydia_Pintscher, MPhamWMF, CBogen, Namenlos314, Gq86, 
Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org