[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Milimetric added a comment. The output is here: https://analytics.wikimedia.org/published/datasets/periodic/reports/metrics/structured-data/ TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

gerritbot added a comment. Change 562555 **merged** by Ottomata: [operations/puppet@production] Enable structured-data report https://gerrit.wikimedia.org/r/562555 TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

gerritbot added a comment. Change 562555 had a related patch set uploaded (by Milimetric; owner: Milimetric): [operations/puppet@production] Enable structured-data report https://gerrit.wikimedia.org/r/562555 TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

gerritbot added a comment. Change 559580 **merged** by Milimetric: [analytics/reportupdater-queries@master] Remove poorly defined metric https://gerrit.wikimedia.org/r/559580 TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

gerritbot added a comment. Change 559580 had a related patch set uploaded (by Milimetric; owner: Milimetric): [analytics/reportupdater-queries@master] Remove poorly defined metric https://gerrit.wikimedia.org/r/559580 TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

gerritbot added a comment. Change 556741 **merged** by Mforns: [analytics/reportupdater-queries@master] Report structured data use for commons https://gerrit.wikimedia.org/r/556741 TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

gerritbot added a comment. Change 556741 had a related patch set uploaded (by Milimetric; owner: Milimetric): [analytics/reportupdater-queries@master] Report structured data use for commons https://gerrit.wikimedia.org/r/556741 TASK DETAIL https://phabricator.wikimedia.org/T239565

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Milimetric added a comment. Ok, seems like some of this confusion is getting cleared up. For my part, here's what I'm planning to do next: - Productionize the query currently getting the 3 million or so `role_name = mediainfo` slots - Productionize the query currently getting the 7

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. > We would still like productionized reports for (3). If that is still possible, I would love to discuss it more :) Please coordinate with #product-analytics on those. I found yet another ticket

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Abit added a comment. Ah sorry about that, the discussion moved to T238878: Data about how many file pages on Commons contain at least one structured data element after I sent that email, and the clarification of the three different definitions I

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. @Abit: Sorry it was not clear. This below is the request you send to analytics couple weeks ago via e-mail, as I mentioned then we rather work on requests via phab tickets that via e-mail. " From: Amanda Bittaker Date: Tue, Nov 19, 2019 at 1:30 AM Subject:

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Abit added a comment. > So, it seems we have 2 completely separate definitions of "structured data" There are three completely separate definitions of structured data: 1. The definition of structured data for the Sloan report 2. The definition of structured data to be reported in

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. > So, it seems we have 2 completely separate definitions of "structured data": Well, the intent of these numbers is not to measure "structured data usage" nor to define that concept. The intent is to measure the impact of the structure data on commons project per

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

matthiasmullie added a comment. No it shouldn’t count pages more than once. UNION omits duplicates (UNION ALL doesn’t), so no need to DISTINCT again. TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Addshore added a comment. SELECT COUNT(*) FROM ( SELECT DISTINCT page_id FROM page INNER JOIN slots ON slot_revision_id = page_latest INNER JOIN content ON slot_content_id = content_id AND content_size > 122 INNER JOIN slot_roles ON role_id =

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. Please see my comment on https://phabricator.wikimedia.org/T238878#5726624 Seems like the 7.9 million items are from contributions of wikidata alone. TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. Adding here public pdf with structure data on commons grant proposal: https://upload.wikimedia.org/wikipedia/foundation/f/f0/Public_Copy_-_Structured_Data_on_Commons_Proposal.pdf TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. @Abit: The queries that report the 7.8 million include , per @matthiasmullie comment both Wikidata Items and Mediainfo items. We can help calculate the percentage of each but from numbers thus far it seems that of those 7.8M items more than half are Wikidata items.

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

mpopov added a comment. @Abit: it's still not entirely clear which query from T238878 @Milimetric should productionize in this ticket. From my conversation with Kate, it seems like your team wants to use the 7.8M number from the Lua-populated

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. @abit: Numbers about SDC will be reported in the platform evolution slides. TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Milimetric, Nuria Cc: Abit, Ramsey-WMF,

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Abit added a comment. @Nuria where will this number be reported in the tuning session? TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Milimetric, Abit Cc: Abit, Ramsey-WMF, kzimmerman,

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. It looks like we are going to have to report this number on the tunning session so taking back my comment above, let's proceed. TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. Let's pause this work as it turns out as there is a parallel effort happening , @Abit to create a ticket for ongoing work TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Abit added a comment. Bless y'all's hearts for setting this up for us ♥♥♥ TASK DETAIL https://phabricator.wikimedia.org/T239565 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Milimetric, Abit Cc: Abit, Ramsey-WMF, kzimmerman, Addshore,

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Nuria added a comment. Some alternatives: superset can source data from other places than druid and we have couple dashboards on top of some tables in staging. This might not be the best option as reportupdater produces tsvs rather than inserting data into staging again. Druid is good for

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

mpopov added a comment. In T239565#5706854 , @Milimetric wrote: > Yay, I get to work with @mpopov :) Aw, I feel likewise! :D > - how often should this report be updated? I think for the intended purpose a monthly

[Wikidata-bugs] [Maniphest] [Commented On] T239565: Create reportupdater reports that execute SDC requests

Milimetric added a comment. Yay, I get to work with @mpopov :) Ok, questions: - how often should this report be updated? - is it exactly that query? This task mentions "queries" plural, just making sure - given the confusion about deletion (T238878#5706835