[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Milimetric]

Milimetric added a comment.


  The output is here: 
https://analytics.wikimedia.org/published/datasets/periodic/reports/metrics/structured-data/

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, 
E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, 
Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, 
Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[gerritbot]

gerritbot added a comment.


  Change 562555 **merged** by Ottomata:
  [operations/puppet@production] Enable structured-data report
  
  https://gerrit.wikimedia.org/r/562555

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, gerritbot
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, 
darthmon_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, 
Salgo60, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, rosalieper, 
Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, 
Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[gerritbot]

gerritbot added a comment.


  Change 562555 had a related patch set uploaded (by Milimetric; owner: 
Milimetric):
  [operations/puppet@production] Enable structured-data report
  
  https://gerrit.wikimedia.org/r/562555

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, gerritbot
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, 
E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, 
Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, 
Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[gerritbot]

gerritbot added a comment.


  Change 559580 **merged** by Milimetric:
  [analytics/reportupdater-queries@master] Remove poorly defined metric
  
  https://gerrit.wikimedia.org/r/559580

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, gerritbot
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, 
darthmon_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, 
Salgo60, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, rosalieper, 
Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, 
Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[gerritbot]

gerritbot added a comment.


  Change 559580 had a related patch set uploaded (by Milimetric; owner: 
Milimetric):
  [analytics/reportupdater-queries@master] Remove poorly defined metric
  
  https://gerrit.wikimedia.org/r/559580

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, gerritbot
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, Nandana, JKSTNK, Akovalyov, Lahi, PDrouin-WMF, Gq86, 
E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, QZanden, Tramullas, 
Acer, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Scott_WUaS, 
Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, Base, aude, 
Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[gerritbot]

gerritbot added a comment.


  Change 556741 **merged** by Mforns:
  [analytics/reportupdater-queries@master] Report structured data use for 
commons
  
  https://gerrit.wikimedia.org/r/556741

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, gerritbot
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, Hook696, Daryl-TTMG, RomaAmorRoma, 0010318400, E.S.A-Sheild, 
darthmon_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, 
JKSTNK, Akovalyov, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, PDrouin-WMF, Gq86, Af420, E1presidente, 
Darkminds3113, Anooprao, SandraF_WMF, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Tramullas, Acer, LawExplorer, 
Salgo60, WSH1906, Lewizho99, Maathavan, Silverfish, _jensen, rosalieper, 
Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, Wikidata-bugs, 
Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, Fabrice_Florin, Raymond, 
Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[gerritbot]

gerritbot added a comment.


  Change 556741 had a related patch set uploaded (by Milimetric; owner: 
Milimetric):
  [analytics/reportupdater-queries@master] Report structured data use for 
commons
  
  https://gerrit.wikimedia.org/r/556741

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, gerritbot
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Milimetric]

Milimetric added a comment.


  Ok, seems like some of this confusion is getting cleared up.  For my part, 
here's what I'm planning to do next:
  
  - Productionize the query currently getting the 3 million or so `role_name = 
mediainfo` slots
  - Productionize the query currently getting the 7 million or so entities 
linked via Lua templates in wbc_entity_usage
  - publish both of these numbers in a single tsv, monthly, updated on the 5th 
of the month
  
  For the future, I suggest looking at rev_timestamp for the revisions/pages 
that are of interest and seeing if it makes sense to get some historical trends 
that way.  See T238878#5730630 
 for some idea of what that 
might look like.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  > We would still like productionized reports for (3). If that is still 
possible, I would love to discuss it more :)
  
  Please coordinate with #product-analytics 
  on those.
  
  I found yet another ticket about SDC metrics in which is apparent that there 
is duplicated work happening. Let's please try to avoid that going forward.
  https://phabricator.wikimedia.org/T231952

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Abit]

Abit added a comment.


  Ah sorry about that, the discussion moved to T238878: Data about how many 
file pages on Commons contain at least one structured data element  
 after I sent that email, and the 
clarification of the three different definitions I listed above came from those 
conversations.
  
  We would still like productionized reports for (3).  If that is still 
possible, I would love to discuss it more :)
  
  >> Matthias is working on (1) in another ticket.
  >
  > Can you link that ticket so we do not duplicate efforts?
  
  It may be T238878  but I'm not 
certain--@matthiasmullie can you please confirm?

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Abit
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  @Abit:  Sorry it was not clear. This below is the request you send to 
analytics couple weeks ago via e-mail, as I mentioned then we rather work on 
requests via phab tickets that via e-mail.
  
  "
  From: Amanda Bittaker 
  Date: Tue, Nov 19, 2019 at 1:30 AM
  Subject: Analytics for structured data elements on Commons
  To: Joseph Allemandou 
  Cc: Ramsey Isler 
  
  We're wondering how many file pages on Commons contain at least one 
structured data element (statement or caption), and we hear you are the person 
to ask.
  
  Is this something you can help us set up analytics for?  If it wouldn't be 
too much effort, there are other structured data numbers it would be nice to 
have, but most urgently we just need to be able to regularly check the number 
of files with structured data.  (We have been using the search results, but 
search indexing is proving unreliable.)   "
  
  > Matthias is working on (1) in another ticket.
  
  Can you link that ticket so we do not duplicate efforts?

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Abit]

Abit added a comment.


  > So, it seems we have 2 completely separate definitions of "structured data"
  
  There are three completely separate definitions of structured data:
  
  1. The definition of structured data for the Sloan report
  2. The definition of structured data to be reported in the Platform Evolution 
tuning session
  3. The definition of structured data for product analytics, such as making 
dashboards and product development choices
  
  I thought that this ticket was about (3).  Separately, Matthias is working on 
(1) in another ticket.  I'm not sure what the status of (2) is.
  
  If this ticket isn't about (3) can you please let me know what you are trying 
to create a report for?

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Abit
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  > So, it seems we have 2 completely separate definitions of "structured data":
  
  Well, the intent of these numbers is not to measure "structured data usage" 
nor to define that concept.  The intent is to measure the impact of the 
structure data on commons project per @Abit initial request couple weeks ago.
  
  With that objective in mind I do not see the Wikidata usage on commons to be 
of relevance as that usage it is not directly tied to the work the team working 
on "structure data on commons" is been doing. Please do let me know if this is 
not  a correct statement.
  
  As a side note I think the true interesting metric is the rate of growth of 
the result of the query below. Following rate of growth of usage of structured 
data on commons growths seems much more relevant to track impact than the 
absolute numbers we have been talking about to date.
  
  SELECT DISTINCT page_id
FROM page
INNER JOIN slots ON slot_revision_id = page_latest
INNER JOIN content ON slot_content_id = content_id AND content_size > 
122
INNER JOIN slot_roles ON role_id = slot_role_id AND role_name = 
'mediainfo'

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[matthiasmullie]

matthiasmullie added a comment.


  No it shouldn’t count pages more than once.
  UNION omits duplicates (UNION ALL doesn’t), so no need to DISTINCT again.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, matthiasmullie
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Addshore]

Addshore added a comment.


SELECT COUNT(*) FROM (
SELECT DISTINCT page_id
FROM page
INNER JOIN slots ON slot_revision_id = page_latest
INNER JOIN content ON slot_content_id = content_id AND content_size > 
122
INNER JOIN slot_roles ON role_id = slot_role_id AND role_name = 
'mediainfo'
UNION
SELECT DISTINCT page_id
FROM page
INNER JOIN wbc_entity_usage ON eu_page_id = page_id
WHERE page_namespace = 6
) AS t;
  
  Is it just me or will this query be counting some pages twice?

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Addshore
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  Please see my comment on https://phabricator.wikimedia.org/T238878#5726624 
Seems like the 7.9 million items are from contributions of wikidata alone.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  Adding here public pdf with structure data on commons grant proposal: 
https://upload.wikimedia.org/wikipedia/foundation/f/f0/Public_Copy_-_Structured_Data_on_Commons_Proposal.pdf

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  @Abit: The queries that report  the 7.8 million include , per @matthiasmullie 
comment both Wikidata Items and Mediainfo items. We can help calculate the 
percentage of each  but from numbers thus far it seems that of those 7.8M items 
more than half are Wikidata items. We can help further quantify that 7M number 
so we have  a precise definition of what we are reporting.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[mpopov]

mpopov added a comment.


  @Abit: it's still not entirely clear which query from T238878 
 @Milimetric should productionize in 
this ticket.
  
  From my conversation with Kate, it seems like your team wants to use the 7.8M 
number from the Lua-populated table using the query from T238878#5683048 
, but there's also an 
overwhelming support for the query in T238878#5708511 
 which yields a count of 3M? 
I've pointed out the problems of missing data and quality in general in the 
Lua-populated table, so I'm not sure if that's the one you want to go with.
  
  Can you or @matthiasmullie please confirm exactly which query should be used?

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, mpopov
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  @abit: Numbers about SDC will be reported in the platform evolution slides.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Abit]

Abit added a comment.


  @Nuria where will this number be reported in the tuning session?

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Abit
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  It looks like we are going to have to report this number on the tunning 
session so taking back my comment above, let's proceed.

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  Let's pause this work as it turns out as there is a parallel effort happening 
 , @Abit to create a ticket for ongoing work

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Abit]

Abit added a comment.


  Bless y'all's hearts for setting this up for us ♥♥♥

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Abit
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Nuria]

Nuria added a comment.


  Some alternatives: superset can source data from other places than druid and 
we have couple dashboards on top of some tables in staging. This might not be 
the best option as reportupdater produces tsvs rather than inserting data into 
staging again. Druid is good for cubes but this is a simple metric that has 1 
dimension so with  the tsvs in mind would be easy to have a dashiki dashboard 
that would look similar to: 
https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os
  
  The dashboard above reports "ratios per browser", if we have 2 metrics:1 ) 
"total_number_of_files" and another with 2) 
total_number_of_files_with_sdc_data/total_number_of_files

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, Nuria
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[mpopov]

mpopov added a comment.


  In T239565#5706854 , 
@Milimetric wrote:
  
  > Yay, I get to work with @mpopov :)
  
  Aw, I feel likewise! :D
  
  > - how often should this report be updated?
  
  I think for the intended purpose a monthly granularity is fine since the 
check-ins have in the past been quarterly or every 6mo. Even if the query takes 
like 35 minutes to run on unsqooped data, would it be okay to schedule it to 
run daily or weekly?
  
  > - is it exactly that query?  This task mentions "queries" plural, just 
making sure
  
  It's starting to look like the query in T238878#5708511 
 is the one that should be 
used?
  
  > - given the confusion about deletion (T238878#5706835 
), should we also count 
stuff from the archive table?
  
  I don't think deleted files should be counted, no.
  
  
  
  I think the end result should be, ideally, a daily-granularity data source in 
Turnilo/Superset having:
  
  - total count of files on Commons
  - total count of files on Commons having structured data (per query in 
T238878#5708511 )
  
  This would enable @Abit & @Ramsey-WMF to track progress of SDC over time in a 
dashboard as (1) an absolute, and (2) relative % (via post-aggregation in 
Superset) in Superset (esp. since that also has periodicity like YoY built in, 
which would be useful for them).
  
  Would have to be careful with the auto aggregation, though. The metrics would 
need to be specified as, like, longMax instead of longSum.
  
  @Milimetric: do you have a destination in mind for the reports? I guess the 
MVP is just a CSV in /srv/published-datasets and we can figure out next steps 
later so this task's scope doesn't blow up, or do y'all have an easy 
pipeline/process for running reportupdater and ingesting the output into Druid?

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric, mpopov
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

[Milimetric]

Milimetric added a comment.


  Yay, I get to work with @mpopov :)  Ok, questions:
  
  - how often should this report be updated?
  - is it exactly that query?  This task mentions "queries" plural, just making 
sure
  - given the confusion about deletion (T238878#5706835 
), should we also count 
stuff from the archive table?

TASK DETAIL
  https://phabricator.wikimedia.org/T239565

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Milimetric
Cc: Abit, Ramsey-WMF, kzimmerman, Addshore, matthiasmullie, gsingers, 
Mayakp.wiki, Ladsgroup, nettrom_WMF, Cparle, Nuria, Milimetric, mpopov, 
4748kitoko, darthmon_wmde, DannyS712, Nandana, JKSTNK, Akovalyov, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Anooprao, SandraF_WMF, GoranSMilovanovic, 
QZanden, Tramullas, Acer, LawExplorer, Salgo60, Silverfish, _jensen, 
rosalieper, Scott_WUaS, Susannaanas, JAllemandou, Jane023, terrrydactyl, 
Wikidata-bugs, Base, aude, Ricordisamoa, Wesalius, Lydia_Pintscher, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs