AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360761
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, Ifrahkhanyaree_WMDE, Manuel, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360761
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, Ifrahkhanyaree_WMDE, Manuel, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360761
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, Ifrahkhanyaree_WMDE, Manuel, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE added a comment.
Prioritizing this now. Initial exploration of the data sources indicates that
we need to use the full `mediawiki_history` rather than
`mediawiki_history_reduced` as the latter doesn't have a distinct
`page_is_deleted` field for Population B.
TASK D
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360761
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, Ifrahkhanyaree_WMDE, Manuel, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE created this task.
AndrewTavis_WMDE added a project: Wikidata.org.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Note that I did this Phabricator tasks search
<https://phabricator.wikimedia.org/search/query/5BIk7a7RSJzT/#R> before making
thi
AndrewTavis_WMDE created this task.
AndrewTavis_WMDE added projects: Mismatch Finder, Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Note that I did the following Phabricator search
<https://phabricator.wikimedia.org/search/query/FxDRSlmcrOEQ/#R> before w
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T362301
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, AndrewTavis_WMDE, luca.favorido, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE created this task.
AndrewTavis_WMDE added projects: Mismatch Finder, Wikidata, wmde-wikidata-tech.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Context
---
A part of the WMDE x Purdue University program where students have been
looking for
AndrewTavis_WMDE renamed this task from "[MSMF] [QB] Remove references of
broken tool from Mismatch Finder and Query Builder" to "[QB] Remove references
of broken tool from Mismatch Finder and Query Builder".
TASK DETAIL
https://phabricator.wikimedia.org/T356659
EMAIL
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T362151
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, Sarai-WMDE, AndrewTavis_WMDE, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T362217
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, AndrewTavis_WMDE, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T362217
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, AndrewTavis_WMDE, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T362217
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, AndrewTavis_WMDE, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1
AndrewTavis_WMDE created this task.
AndrewTavis_WMDE added projects: Mismatch Finder, Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
**Steps to replicate the issue**:
In looking at the mismatches on Mismatch Finder
<https://mismatch-finder.toolforge.
AndrewTavis_WMDE created this task.
AndrewTavis_WMDE added projects: Mismatch Finder, Wikidata, Wikidata Dev Team.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Problem
---
Upon uploading some new mismatches, something that I'm realizing is tha
AndrewTavis_WMDE added a comment.
Note that in checking the `tmp` directory just now, there still are
files/directories in there, meaning that parts of the process are likely still
running (maybe parts that don't need private data access). We'll be checking
this again in a mont
AndrewTavis_WMDE renamed this task from "[Analytics] Monthly repeating tasks
(next: March 2024)" to "[Analytics] Monthly repeating tasks (next: April 2024)".
TASK DETAIL
https://phabricator.wikimedia.org/T342559
EMAIL PREFERENCES
https://phabricator.wikimedi
AndrewTavis_WMDE added a comment.
Sheet has been updated for March via a query of
`wmde.wd_rest_api_metrics_monthly` that's generated by Airflow. Slightly lower
user agents than last month, but IPs doubled 📈
TASK DETAIL
https://phabricator.wikimedia.org/T342559
EMAIL PREFERENCES
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T342559
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Manuel, Aklapper, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T361203
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Manuel, Aklapper, AndrewTavis_WMDE, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356659
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, karapayneWMDE, AndrewTavis_WMDE, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE added a comment.
Updated the description as wmde/wikidata-mismatch-finder#878
<https://github.com/wmde/wikidata-mismatch-finder/pull/878> fixed the problem
for Mismatch Finder. At time of writing Curious Facts is still referenced in
the Query Builder footer.
TASK
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356659
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, karapayneWMDE, AndrewTavis_WMDE, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356618
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Michael, karapayneWMDE, Aklapper, Manuel, AndrewTavis_WMDE
AndrewTavis_WMDE lowered the priority of this task from "Medium" to "Low".
TASK DETAIL
https://phabricator.wikimedia.org/T342559
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: AndrewTavis_WM
AndrewTavis_WMDE added a comment.
I've added the numbers for February to the sheet based on the first DAG run
and also just went through the query job one final time to check. The queries
that are being ran by the job are directly from the original queries with only
a few minor ch
AndrewTavis_WMDE added a comment.
Note that this task is dependent on whether a standardized system that would
not require the published datasets is created. Such a system is discussed in
T361214: Public dashboard process <https://phabricator.wikimedia.org/T361214>.
TASK DETAIL
AndrewTavis_WMDE removed AndrewTavis_WMDE as the assignee of this task.
TASK DETAIL
https://phabricator.wikimedia.org/T361203
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Manuel, Aklapper, AndrewTavis_WMDE
AndrewTavis_WMDE added a comment.
Note that I've made T361214: Public dashboard process
<https://phabricator.wikimedia.org/T361214> to explain our use case of a
standardized public dashboard process :)
TASK DETAIL
https://phabricator.wikimedia.org/T360298
EMAIL PREFEREN
AndrewTavis_WMDE created this task.
AndrewTavis_WMDE added projects: Wikidata Analytics (Kanban), Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
In T341330: [Analytics] Airflow implementation of unique ips accessing
Wikidata's REST API metrics &
AndrewTavis_WMDE added a comment.
Merge request
<https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/631>
has been brought in, and we've successfully deployed! 🎉 An output from the new
`wmde.wd_rest_api_metrics_monthly` table is:
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T341330
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Manuel, Aklapper, Danny_Benjafield_WMDE, S8321414
AndrewTavis_WMDE renamed this task from " [Analytics] Public Superset dashboard
pilot" to " [Analytics] Public dashboard pilot".
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360298
EMAIL PREFERENCES
https://phabricator.wi
AndrewTavis_WMDE added a comment.
Post a large discussion about this in the `data-engineering-collab` channel
on Slack, the general findings for this are:
- The public Superset instance isn't suitable for this at this time and
there's no time table for it to be (see abov
AndrewTavis_WMDE added a comment.
Further checks on this: the dashboarding process for the public Superset
seems to be based on a few preset databases that have the data from Wikimedia
projects (see SQL Lab <https://superset.wmcloud.org/sqllab/>). As of now I'm
doubting whether w
AndrewTavis_WMDE added a comment.
Note that from the most recent discussions with WMF data engineering, there
isn't a set workflow for getting information into a place where it can be
accessed via the Public Superset instance. We would need to edit the DAG such
that we include an e
AndrewTavis_WMDE added a comment.
Exciting! I'll play around a bit towards the end of next week and send along
a PR with the workflow, docs and changes given the local run warnings 😊 Will
let you know if anything comes up before then. Have a nice weekend when it
comes along!
TASK D
AndrewTavis_WMDE claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T360761
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, Ifrahkhanyaree_WMDE, Manuel, Danny_Benjafield_WMDE,
Astuthiodit_1, karapayneWMDE
AndrewTavis_WMDE added a comment.
@nshahquinn-wmf, @xcollazo: checking in on this one again. I would have some
time in the next two weeks or so to implement a PR workflow check of linting
and code formatting. If folks are fine with Ruff
<https://github.com/astral-sh/ruff> that'd
AndrewTavis_WMDE added a comment.
Merge request for this has been sent and can be found here
<https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/631>
:) Requested WMF's review on this first one, but we'll need to take over from
there unless th
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T341330
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Manuel, Aklapper, Danny_Benjafield_WMDE, Astuthiodit_1
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356618
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Michael, karapayneWMDE, Aklapper, Manuel, AndrewTavis_WMDE
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356618
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Michael, karapayneWMDE, Aklapper, Manuel, AndrewTavis_WMDE
AndrewTavis_WMDE closed this task as "Resolved".
AndrewTavis_WMDE claimed this task.
AndrewTavis_WMDE added a comment.
Fantastic! Thank you both again for the help here :) Really is great to be
winding down these processes and moving onto the next steps! 🎉
TASK DETA
AndrewTavis_WMDE added a comment.
Thank you both so much! Let me know when the GitHub repos have been deleted
and I'll resolve this and update the greater epic 😊
TASK DETAIL
https://phabricator.wikimedia.org/T357697
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/
AndrewTavis_WMDE edited projects, added Wikidata Analytics (Kanban); removed
Wikidata Analytics.
TASK DETAIL
https://phabricator.wikimedia.org/T357697
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: hashar, brouberol, Manuel
AndrewTavis_WMDE added a comment.
Per suggestion from @noarave I reran the curl command
<https://github.com/wmde/wikidata-mismatch-finder/blob/main/docs/UserGuide.md#example-with-curl>
with `-v` at the end for a verbose output. Of note is in the first line we
have `Note: Unnecessary
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360436
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: noarave, Aklapper, AndrewTavis_WMDE, Danny_Benjafield_WMDE, Astuthiodit_1
AndrewTavis_WMDE added a project: Wikidata Dev Team.
TASK DETAIL
https://phabricator.wikimedia.org/T351072
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, Lucas_Werkmeister_WMDE, AndrewTavis_WMDE, Michael, Manuel, georg
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360436
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: noarave, Aklapper, AndrewTavis_WMDE, Danny_Benjafield_WMDE, Astuthiodit_1
AndrewTavis_WMDE added a subtask: T360436: [MSMF] Add upload file limit to
Mismatch Finder documentation.
TASK DETAIL
https://phabricator.wikimedia.org/T349816
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, ItamarWMDE
AndrewTavis_WMDE added a parent task: T349816: [EPIC] [MSMF] Consolidate
documentation.
TASK DETAIL
https://phabricator.wikimedia.org/T360436
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: noarave, Aklapper, AndrewTavis_WMDE
AndrewTavis_WMDE created this task.
AndrewTavis_WMDE added projects: wmde-wikidata-tech, Mismatch Finder, Wikidata.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
Context
---
In the Mismatch Finder User Guide
<https://github.com/wmde/wikidata-mismatch-fin
AndrewTavis_WMDE claimed this task.
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360298
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE added a comment.
Thanks! I'll give an estimate on the timing of this once we've finished up
T341330: [Analytics] Airflow implementation of unique ips accessing Wikidata's
REST API metrics <https://phabricator.wikimedia.org/T341330>. I'll need
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360296
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Michael, ECohen_WMDE, Aklapper, Pamputt, AndrewTavis_WMDE, JeanFred
AndrewTavis_WMDE claimed this task.
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T360296
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Michael, ECohen_WMDE, Aklapper, Pamputt
AndrewTavis_WMDE added a comment.
We should discuss what the frequency of the jobs we're discussing is:
- For the Wiktionary Cognate data to be of use to editors I'd say we'd be
looking at daily jobs as a user would want to be able to make needed edits and
then check the
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T358254
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: ECohen_WMDE, AndrewTavis_WMDE, Manuel, Aklapper, Danny_Benjafield_WMDE
AndrewTavis_WMDE added a comment.
Moving on to the Usage Dashboard, what it is we're looking for is the
following two tables:
| Project | Project Type | Total Articles | Percent Articles Using WD | Total
Articles Using WD | Percent Articles With Sitelinks | Total Articles
AndrewTavis_WMDE added a comment.
Looking into this more, I'm as of now not sure how the original connection to
the Cognate extension data was made. I'm seeing no inputs from a source
database in the Wiktionary Cognate dashboard code. The server is loading in
data from the
AndrewTavis_WMDE added a comment.
Initial explorations of the Wiktionary directory
<https://analytics.wikimedia.org/published/datasets/wmde-analytics-engineering/Wiktionary/>
indicate that the data we're trying to replicate here is in projectData
<https://analytics.wikimedia
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T358254
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: ECohen_WMDE, AndrewTavis_WMDE, Manuel, Aklapper, Danny_Benjafield_WMDE
AndrewTavis_WMDE claimed this task.
TASK DETAIL
https://phabricator.wikimedia.org/T358254
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: ECohen_WMDE, AndrewTavis_WMDE, Manuel, Aklapper, Danny_Benjafield_WMDE,
Astuthiodit_1
AndrewTavis_WMDE added a comment.
Hey @taavi 👋 One thing to note is that a decision was made to deprecate the
processes that are running completely. WMDE analytics has no plan of
maintaining `quratorqcerevolver` or `quratorqcfrevolver` - also known as
Current Events (ce) and Curious Facts
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356618
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: hashar, Michael, karapayneWMDE, Aklapper, Manuel, AndrewTavis_WMDE
AndrewTavis_WMDE added projects: Projects-Cleanup,
Continuous-Integration-Config, Wikimedia-GitHub, wmde-wikidata-tech.
TASK DETAIL
https://phabricator.wikimedia.org/T357697
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Manuel
AndrewTavis_WMDE added a comment.
Updated the above comment with a second run and also ran a query for the
total IPs for the given period, with the result being `2,115,166`. Percent
Scholia queries for the period is thus `28918 / 2115166 * 100`, or 1.37%.
TASK DETAIL
https
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE added a comment.
A follow up request from @Manuel on this was for the total IPs that are
accessing Scholia. The following query was run for this:
SELECT
count(
DISTINCT CASE
WHEN query LIKE '%# tool: scholia%' THEN http
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE moved this task from Prioritized backlog to Product
verification on the Wikidata Analytics (Kanban) board.
AndrewTavis_WMDE added a comment.
Credit on checking the queries goes to @dcausse :) Added the percent that are
identified via a user agent to the results summary just
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE renamed this task from "Archive WMDE analytics repositories"
to "Archive WMDE analytics Gerrit repositories".
TASK DETAIL
https://phabricator.wikimedia.org/T357697
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To:
AndrewTavis_WMDE created this task.
AndrewTavis_WMDE added projects: Wikidata, Wikidata Analytics.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
This task is based on T354534: Archive Wikidata Concepts Monitor repositories
<https://phabricator.wikimedia.org/T354
AndrewTavis_WMDE added a comment.
Note that this task will include the `user_agent` values as well as we'll be
doing the typical reporting metrics in one query. As of now we had three
different functions being ran, but this can be simplified to one HiveQL process
that then updates a
AndrewTavis_WMDE moved this task from Monitoring to Kanban on the Wikidata
Analytics board.
AndrewTavis_WMDE edited projects, added Wikidata Analytics (Kanban); removed
Wikidata Analytics.
TASK DETAIL
https://phabricator.wikimedia.org/T341330
WORKBOARD
https://phabricator.wikimedia.org
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356618
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: hashar, Michael, karapayneWMDE, Aklapper, Manuel, AndrewTavis_WMDE
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE added a comment.
Having derived quick samples (`DISTRIBUTE BY rand()` to mix it up, but
nothing more), what I'm seeing is that the comment queries look to be very
similar to one another regardless of if they're spiders or non-spiders. Could
be that what we're
AndrewTavis_WMDE added a comment.
Quick counts as in the sampling task to check uniqueness of queries and HTTP
statuses (I don't think that other measures like variance over weeks, duration
or char size would add much). Note that percentages below are for the
sub-groups, not for all Sc
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356659
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Aklapper, karapayneWMDE, AndrewTavis_WMDE, Danny_Benjafield_WMDE,
Astuthiodit_1
AndrewTavis_WMDE added a comment.
Results from the following query to check automate traffic via isSpiderUDF
<https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-hive/src/main/java/org/wikimedia/analytics/refinery/hive/IsSpiderUDF.java>
is that `91.36%` of the
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356618
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: hashar, Michael, karapayneWMDE, Aklapper, Manuel, AndrewTavis_WMDE
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T356618
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: hashar, Michael, karapayneWMDE, Aklapper, Manuel, AndrewTavis_WMDE
AndrewTavis_WMDE added a comment.
Sheet has been updated with the numbers for January. Note that we have less
unique user agents from a local maximum last month, but the number of IPs
continues to grow. Seems like adoption is picking up, but then we're not
necessarily going to pick th
AndrewTavis_WMDE renamed this task from "[Analytics] Monthly repeating tasks
(next: February 2024)" to "[Analytics] Monthly repeating tasks (next: March
2024)".
AndrewTavis_WMDE changed the task status from "In Progress" to "Stalled".
AndrewTavis_WMDE up
AndrewTavis_WMDE changed the task status from "Open" to "In Progress".
AndrewTavis_WMDE triaged this task as "Medium" priority.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailprefere
AndrewTavis_WMDE changed the status of subtask T353453: [Analytics] Impact of
Scholia on WDQS from "Open" to "In Progress".
TASK DETAIL
https://phabricator.wikimedia.org/T337799
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpref
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE added a comment.
Here are some initial results for consideration. Using the following query
over the full dataset from `event.wdqs_external_sparql_query` (last 90 days):
SELECT
count(*) AS total_scholia_queries
FROM
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
AndrewTavis_WMDE added a comment.
Quick note on this:
There are two ways that need to be factored in to deriving if a query is from
Scholia. Some queries do start with `#tool: scholia` as @dcausse suggested, but
I checked for user agents and also found that the string `"Scholia
AndrewTavis_WMDE added a comment.
Task is refined and I'm starting work on it now. I'm assuming that
`event.wdqs_external_sparql_query` is what I'd use for this, and thus we'd be
getting aggregate/percent values within a 90 day period given the retention
policy :)
AndrewTavis_WMDE updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T353453
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AndrewTavis_WMDE
Cc: Lydia_Pintscher, dcausse, Aklapper, Manuel, Danny_Benjafield_WMDE
101 - 200 of 628 matches
Mail list logo
