| GoranSMilovanovic added a subscriber: Addshore. GoranSMilovanovic added a comment. |
@Addshore I need a second opinion on the following, please. One of your generating scripts for this Grafana dashboard iterates across the project databases and counts the pages that make use of any aspects except 'S', in the following manner:
SELECT COUNT(DISTINCT eu_page_id) AS pages FROM dewiki.wbc_entity_usage WHERE eu_aspect != 'S';
Only the dewiki example shown above takes 3 minutes and 20 seconds from analytics-store.eqiad.wmnet to complete (run from stat1005), and there are >800 projects to assess in this manner.
Suggestion. I have all these SQL data sqooped in a Hive goransm.wdcm_clients_wb_entity_usage table. This table has a weekly update (i.e. there is a weekly Apache Sqoop run across all SQL wbc_entity_usage tables that collects them to Hadoop), but let's say I can make that daily. If you can help me around the Graphite metrics, only in terms of helping me understand the proper conventions for metric names, I think I could start sending these data to Graphite on daily basis from R in production. That would (a) save some resources on our SQL servers, and (b) align us to the general policy of bypassing SQL wherever we deal with real Big Data sets. Let me know what you think.
@Lydia_Pintscher I think it is natural to approach this task within the WDCM framework, but that would imply presenting the outputs on an RStudio Shiny Server (like all WDCM dashboards), not Grafana. Let me know if this has to be Grafana for some reason. Otherwise I can easily build you a dashboad for this an any other similar, related statistics in WDCM.
Cc: Addshore, Aklapper, Lydia_Pintscher, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
