subject:"\[Wikidata\-bugs\] \[Maniphest\] T336361\: \[Analytics\] Identify access via mobile devices"

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-09-01 Thread Manuel

Manuel added a comment.


  > Here are the tables that break down the device_family values, @Manuel :)
  
  Wow, that's great, thank you! It is even more insightful than I thought! 
  
  It seems that with `user_agent_map.os_family = 'Android', iOS or KaiOS` we 
can now reliably identify mobile devices. \o/
  
  However, my assumption is that when only filtering for `agent_type != 
'spider'` the population will still include a lot of non-UI hits. This means 
that we would likely over-estimate desktop devices.
  
  Suggestion for action after our vacation:
  
  - Look into identifying desktop hits more, also looking into @Michael 's 
suggestions (T336361#9080129 
)
  
  Let's talk about this more after you are back, have a great vacation! :)
  
  (//P.S. I have copied your other two comments to T344052: [Analytics] Join 
information from access-tables (like webrequest) with edit-tables tables (that 
include oldids)   to keep the issues 
separated. I'll answer there.) //

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-09-01 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-09-01 Thread AndrewTavis_WMDE

AndrewTavis_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-09-01 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-09-01 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-09-01 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-31 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  As far as totals for this task are concerned, @Manuel, what I'm getting is 
the following:
  
  **Population for the following is**:
  
  All views of `www.wikidata.org` and `m.wikidata.org` inclusively between 
01/7/2023 and 31/7/2023 that are further `agent_type != 'spider'` and mobile 
users with `user_agent_map.os_family = 'Android'`, `iOS` or `KaiOS`.
  
WITH pageview_actor_os_families AS (
SELECT
user_agent_map.os_family AS os_family

FROM 
wmf.pageview_actor

WHERE
year = 2023
AND month = 7
AND '2023-07-01' <= dt
AND dt < '2023-08-01'
AND uri_host IN ('www.wikidata.org', 'm.wikidata.org')
AND is_pageview = True
AND agent_type = 'user'
)

SELECT
COUNT(*) AS total_views

FROM
pageview_actor_os_families

WHERE
os_family = 'Android'
OR os_family = 'iOS'
OR os_family = 'KaiOS'
  
  ... gives us `699,968`. For edits I need some help filtering out bots as we 
don't have the option of going from webrequests as mentioned above and I'm 
having a hard time figuring out how to access MediaWiki tables directly for 
`user_groups`. With that being said, I currently have the following that we 
just need to get the bots out of:
  
WITH cu_changes_os_families AS (
SELECT
ua_parser(cuc_agent) AS user_agent_map

FROM 
wmf_raw.mediawiki_private_cu_changes

WHERE
wiki_db = 'wikidatawiki'
AND '20230701' <= cuc_timestamp
AND cuc_timestamp < '20230801'
)

SELECT
COUNT(*) AS total_edits

FROM
cu_changes_os_families

WHERE
user_agent_map.os_family = 'Android'
OR user_agent_map.os_family = 'iOS'
OR user_agent_map.os_family = 'KaiOS'
  
  ... gives us `221,073`. Let me know if you have any thoughts about how to 
make the connection to the MediaWiki tables or if not also where we could get 
bot status from! :)

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-31 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  @Manuel, I think we can throw out the idea of creating an edits subset of 
webrequests, sadly :( The following would be where we'd find the various 
actions that we'd need to collect to define as edits fully: 
https://www.wikidata.org/w/api.php. We know at the very least that we'd want 
`uri_query LIKE '?action=edit%'` and `uri_query LIKE '?action=wbsetclaim%'`, 
but figuring out what else needs to be added seems to be prohibitive given the 
discrepancy:
  

SELECT
COUNT(*) AS total_edits

FROM 
wmf.webrequest

WHERE
year = 2023
AND month = 7
AND day = 31
AND uri_host IN ('www.wikidata.org', 'm.wikidata.org')
AND (
uri_query LIKE '?action=edit%'
OR uri_query LIKE '?action=wbsetclaim%'
)
  
  ... gives us `11,947`, and the following:
  
SELECT
COUNT(*) AS total_edits

FROM 
wmf_raw.mediawiki_private_cu_changes

WHERE
wiki_db = 'wikidatawiki'
AND month = '2023-07'
AND '20230731' <= cuc_timestamp
AND cuc_timestamp < '20230801'
  
  ... gives us `657,347`, with `11947/657347` being `1.817%`. There definitely 
should be a combination of those actions that gets us a similar number, but 
this would be something that we'd need to loop WMF into, and the easiest route 
would likely be to talk about getting a similar subset to `pageview_actor` as a 
table in the Data Lake.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-31 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  Here are the tables that break down the `device_family` values, @Manuel :) As 
before:
  
  **Population for the following is**:
  
  All requests  to `www.wikidata.org` and `m.wikidata.org` inclusively between 
25/7/2023 and 31/7/2023 that are further `agent_type != 'spider'`.
  
  Android Desktop
  ---
  
  Crazy long tail on this one that exceeded expectations. `K` is Chrome as 
detailed in this Medium article 

 (they were seeing hits from K-Kool, which was Google Analytics assuming that 
the `K` for Chrome was the next alphabetical device). Here's the post from 
Google about the shift to the new `user_agent` structure:
  
  
https://developer.chrome.com/blog/user-agent-reduction-android-model-and-version/
  
  | device_family| total_requests | percent |
  |  | -- | --- |
  | K| 3026711  | 29.54   |
  | Samsung GT-I9100 | 765508 | 7.4712  |
  | Generic Smartphone   | 622962 | 6.08|
  | Nexus 5  | 481815 | 4.7024  |
  | Pixel 2  | 478835 | 4.6733  |
  | Samsung SM-G900P | 477533 | 4.6606  |
  | Samsung Galaxy Nexus | 262219 | 2.5592  |
  | Samsung SM-A515F | 251317 | 2.4528  |
  | Samsung SM-A217M | 231445 | 2.2589  |
  | Samsung SM-G960U | 118989 | 1.1613  |
  | Samsung SM-A102U | 113991 | 1.1125  |
  | Samsung SM-A205U | 106258 | 1.0371  |
  |
  
  
  
  Android Mobile
  --
  
  Not as spread to the tail, with most of the values being now in `K` which is 
for Chrome.
  
  | device_family| total_requests | percent |
  |  | -- | --- |
  | K| 8500919  | 42.8627 |
  | Nexus 5  | 991174 | 4.9976  |
  | Pixel 2  | 985724 | 4.9701  |
  | Samsung SM-G900P | 983740 | 4.9601  |
  | Samsung GT-I9100 | 896721 | 4.5214  |
  | Generic Smartphone   | 741579 | 3.7391  |
  | Samsung SM-A515F | 332731 | 1.6777  |
  | Samsung Galaxy Nexus | 309796 | 1.562   |
  | Samsung SM-A217M | 301320 | 1.5193  |
  | Samsung SM-G960U | 157173 | 0.7925  |
  | Samsung SM-A102U | 152035 | 0.7666  |
  | Samsung SM-A205U | 144086 | 0.7265  |
  |
  
  
  
  iOS Desktop
  ---
  
  There definitely are more iPads that are using the desktop view. In iOS you 
can directly request the desktop version on a website, and there could also be 
settings for this (or they could have saved the desktop version to their 
bookmarks).
  
  | device_family | total_requests | percent |
  | - | -- | --- |
  | iPhone| 3116794  | 49.9937 |
  | iPad  | 3029248  | 48.5894 |
  | iOS-Device| 73606  | 1.1806  |
  | Other | 5848   | 0.0938  |
  | Mac   | 5388   | 0.0864  |
  | iPod  | 3494   | 0.056   |
  |
  
  
  
  iOS Mobile
  --
  
  iOS Device, Mac and Other are lower in this one. My idea for iPods was 
actually wrong as there are more using in the desktop table above. More 
requests from iPhones and iPads in aggregate, but percentage wise more iPhones.
  
  | device_family | total_requests | percent |
  | - | -- | --- |
  | iPhone| 7499220  | 64.8608 |
  | iPad  | 4026622  | 34.8263 |
  | iOS-Device| 33574  | 0.2904  |
  | iPod  | 2572   | 0.0222  |
  | Mac   | 32 | 3.0E-4  |
  | Other | 3  | 0.0 |
  |
  
  
  
  Tizen Desktop
  -
  
  It actually is smart TVs! 
  
  | device_family| total_requests | percent |
  |  | -- | --- |
  | Samsung SMART-TV | 10540  | 96.9285 |
  | Samsung Family   | 300| 2.7589  |
  | Samsung SM-Z200Y | 20 | 0.1839  |
  | Samsung SM-Z400Y | 6  | 0.0552  |
  | Samsung SM-R840  | 4  | 0.0368  |
  | Samsung SM-Z200F | 2  | 0.0184  |
  | Samsung SM-R845F | 1  | 0.0092  |
  | Samsung SM-R810  | 1  | 0.0092  |
  |
  
  
  
  Tizen Mobile
  
  
  Samsung Family is a smart fridge :) I think we're good to not include Tizen 
in mobile devices.
  
  | device_family| total_requests | percent |
  |  | -- | --- |
  | Samsung Family   | 447| 61.4856 |
  | Samsung SM-Z200Y | 163| 22.4209 |
  | Samsung SMART-TV | 45 | 6.1898  |
  | Samsung SM-Z200F | 26 | 3.5763  |
  | Samsung SM-Z400Y | 24 | 3.3012  |
  | Samsung SM-R810  | 22

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-31 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a subscriber: JAllemandou.
AndrewTavis_WMDE added a comment.


  And here are the finalized heuristics (@JAllemandou, tagging you as well). 
The following query is saved as a temporary view as `df_requests_subset`:
  
SELECT
user_agent_map.os_family AS os_family,
user_agent_map.device_family AS device_family,
access_method AS access_method

FROM 
wmf.webrequest

WHERE
year = 2023
AND month = 7
AND '2023-07-25' <= dt
AND dt < '2023-08-01'
AND uri_host IN ('www.wikidata.org', 'm.wikidata.org')
AND agent_type != 'spider'
  
  The total requests for the given week is `217,406,068`.
  
  OS Family
  -
  
  | os_family | total_requests | percent_total |
  | - | -- | - |
  | Other | 75242990  | 34.6094   |
  | Windows   | 58995336  | 27.136|
  | Android   | 31297765  | 14.396|
  | Mac OS X  | 22126542  | 10.1775   |
  | iOS   | 19452224  | 8.9474|
  | Linux | 8584799  | 3.9487|
  | Ubuntu| 1114790  | 0.5128|
  | Chrome OS | 440958 | 0.2028|
  | Fedora| 103886 | 0.0478|
  | Tizen | 11601  | 0.0053|
  | KaiOS | 7206   | 0.0033|
  | Chromecast| 5738   | 0.0026|
  | FreeBSD   | 5585   | 0.0026|
  | OpenBSD   | 4023   | 0.0019|
  | Windows Phone | 2997   | 0.0014|
  |
  
  
  
  Windows OS Family Access Methods
  
  
  | access_method | total_requests | percent |
  | - | -- | --- |
  | desktop   | 57528793  | 97.5141 |
  | mobile web| 1466543  | 2.4859  |
  |
  
  
  
  Mac OS X and Mac OS OS Family Access Methods
  
  
  | access_method | total_requests | percent |
  | - | -- | --- |
  | desktop   | 21498169  | 97.1598 |
  | mobile app| 5042   | 0.0228  |
  | mobile web| 623409 | 2.8175  |
  |
  
  
  
  Android OS Family Access Methods
  
  
  | access_method | total_requests | percent |
  | - | -- | --- |
  | desktop   | 10246138  | 32.7376 |
  | mobile app| 1218698  | 3.8939  |
  | mobile web| 19832929  | 63.3685 |
  |
  
  
  
  iSO OS Family Access Methods
  
  
  | access_method | total_requests | percent |
  | - | -- | --- |
  | desktop   | 6234378  | 32.0497 |
  | mobile app| 1655823  | 8.5123  |
  | mobile web| 11562023  | 59.4381 |
  |
  
  
  
  KaiOS OS Family Access Methods
  --
  
  | access_method | total_requests | percent |
  | - | -- | --- |
  | desktop   | 976| 13.5443 |
  | mobile web| 6230   | 86.4557 |
  |
  
  
  
  Device Family
  -
  
  | device_family| total_requests | percent_total |
  |  | -- | - |
  | Other| 146058185  | 67.1822   |
  | Mac  | 22138790  | 10.1832   |
  | K| 11527630  | 5.3023|
  | iPhone   | 10638696  | 4.8935|
  | iPad | 7057060  | 3.246 |
  | Generic Smartphone   | 1691215  | 0.7779|
  | Samsung GT-I9100 | 1662229  | 0.7646|
  | Nexus 5  | 1473452  | 0.6777|
  | Pixel 2  | 1465517  | 0.6741|
  | Samsung SM-G900P | 1461273  | 0.6721|
  | Samsung SM-A515F | 593669 | 0.2731|
  | Samsung Galaxy Nexus | 572015 | 0.2631|
  | Samsung SM-A217M | 534056 | 0.2456|
  | Samsung SM-G960U | 277306 | 0.1276|
  | Samsung SM-A102U | 266371 | 0.1225|
  | Samsung SM-A205U | 250568 | 0.1153|
  | M2006C3MT| 179344 | 0.0825|
  | XiaoMi HM NOTE 1LTEW | 161811 | 0.0744|
  | Samsung SM-G991U | 150001 | 0.069 |
  | Samsung SM-A325M | 144782 | 0.0666|
  | Samsung SM-A305GT| 140535 | 0.0646|
  | Samsung SM-G960F | 137910 | 0.0634|
  | Oppo CPH1931 | 137533 | 0.0633|
  | Samsung SM-G950F | 137498 | 0.0632|
  | Oppo CPH1877 | 134059 |

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-31 Thread Manuel

Manuel added a comment.


  > I am now not sure anymore if my assumption was right that Tizen is mostly 
older mobile phones. Wouldn't we then expect a desktop/mobile ratio more 
similar to Android?
  
  Idea: They are likely misidentified by our server and that is why the desktop 
interface is delivered.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-30 Thread Manuel

Manuel added a comment.


  I am now not sure anymore if my assumption was right that Tizen is mostly 
older mobile phones. Wouldn't we then expect a desktop/mobile ratio more 
similar to Android?
  
  Could you please investigate this combination more (and all others that seem 
suspicios)? We can focus only on stuff that might have an impact at all when 
comparing device-type (mobile vs desktop) vs UI used (m vs. www).
  
  To check what combinations are even relevant (not sure about Tizen, as we are 
only talking about 12K hits) we could check if there are substantial 
differences between the Global South and Global North. I will open a separate 
task for this.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-30 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  Here are the values for `Tizen` as well, @Manuel:
  
  | access_method | total_requests | percent |
  | - | -- | --- |
  | desktop   | 10874  | 93.7333 |
  | mobile web| 727| 6.2667  |
  |
  
  I'd assume that the Smart TV accesses are being logged as `desktop`, and the 
`mobile web` values are older devices.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: JAllemandou, AndrewTavis_WMDE, Michael, Manuel, Aklapper, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-30 Thread Manuel

Manuel added a comment.


  > Tizen
  
  We will use this data to work to get answers to some diversity questions. So 
it would be important to ensure that we do not systematically exclude 
non-premium smartphone devices. I know that e.g. Samsung used Tizen on their 
non-premium smarphones. So this might be what we are seeing here. KaiOS sounds 
like a good candidate as well.
  
  Ideally we would evaluate the list again from this perspective to make sure 
that we do not accidentally introduce bias in the data.
  
  > gent_type != 'spider'
  
  That list is not what I would have expected, so I investigated some more:
  
  The spider/user classification recorded in the agent_type field is based on 
the user agent, evaluated against some regexes 
. Based on the 
code my assumption would be that the Android and iOS user agents that we are 
seeing are crawlers that are accessing the mobile page via some sort of mobile 
user agent.
  
  It is important to me that we always have a good understanding of what 
exactly we are looking at:
  
  So, yes, let's exclude spiders (we are only interested in UI views and edits 
by humans) but make sure that we document the resulting base well when we show 
results: "The population conisits of all accesses to www.wikidata.org and 
m.wikidata.org between XXX and XXX, excluding | WMF-identified 
 spiders." (or 
something similar :).

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-30 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  We'd talked "Tizen" a bit this morning, @Manuel, but let's not focus on it. 
Did a bit of Wikipedia research  and since 
since 2021 it's mostly in use in Samsung Smart TVs. That leaves us with 
`Android` and `iOS` for the predominant mobile `os_family` values, and if we 
want to include a Linux based one it'd be `KaiOS`.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-30 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  Here are the answers to the three questions we had from the daily, @Manuel:
  
  `uri_host LIKE 'wikidata'` as mentioned above:
  
  | uri_host   |
  | -- |
  | test.m.wikidata.org|
  | query-preview.wikidata.org |
  | m.wikidata.org |
  | www.m.wikidata.org |
  | query.wikidata.org |
  | test.wikidata.org  |
  | www.wikidata.org   |
  | wikidata.org   |
  |
  
  Results for the following:
  
SELECT
DISTINCT agent_type AS agent_type

FROM 
wmf.webrequest

WHERE
year = 2023
AND month = 7
AND '2023-07-25' <= dt
AND dt < '2023-08-01'
  
  
  
  | agent_type |
  | -- |
  | spider |
  | user   |
  |
  
  And `os_family`/`device_family` results for `agent_type = 'spider'` via the 
following query:
  
SELECT
user_agent_map.os_family AS os_family,
user_agent_map.device_family AS device_family,
COUNT(*) AS total_requests,
ROUND(COUNT(*) / CAST( SUM(COUNT(*)) OVER () AS float) * 100, 4) AS 
percent_total

FROM 
wmf.webrequest

WHERE
uri_host IN ('www.wikidata.org', 'm.wikidata.org')
AND agent_type = 'spider'

GROUP BY
os_family,
device_family

ORDER BY
total_requests DESC
  
  
  
  | os_family | device_family| total_requests | percent_total |
  | - |  | -- | - |
  | Other | Spider   | 139680267  | 47.6981   |
  | Other | Other| 132010385  | 45.079|
  | Android   | Spider   | 10371644  | 3.5417|
  | Android   | Generic Smartphone   | 7175211  | 2.4502|
  | Other | Generic Feature P... | 2658042  | 0.9077|
  | Mac OS X  | Mac  | 450696 | 0.1539|
  | iOS   | Spider   | 155612 | 0.0531|
  | Linux | Other| 101342 | 0.0346|
  | Android   | Samsung SM-G930V | 84047  | 0.0287|
  | Windows   | Spider   | 72628  | 0.0248|
  | Linux | Spider   | 45399  | 0.0155|
  | Windows   | Other| 14062  | 0.0048|
  | Other | Mac  | 8591   | 0.0029|
  | iOS   | iPhone   | 5062   | 0.0017|
  | Gentoo| Spider   | 1234   | 4.0E-4|
  | Android   | XIG02| 737| 3.0E-4|
  | iOS   | iPad | 680| 2.0E-4|
  | Android   | Samsung SM-G570F | 533| 2.0E-4|
  | - | -| 471| 2.0E-4|
  | Android   | SC-53C   | 148| 1.0E-4|
  |
  
  All things considered, I think we're good to do `agent_type != 'spider'` :)

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-30 Thread Manuel

Manuel added a comment.


  Hi Andrew, thx for the outputs that we're getting for LIKE 'wikidata' !
  
  Wikidata (desktop interface)
  
  - 'www.wikidata.org'
  - 'wikidata.org' (should only be a redirect to the main URI)
  
  Wikidata (mobile interface)
  
  - 'm.wikidata.org'
  
  Test Wikidata
  
  - 'test.m.wikidata.org'
  - 'test.wikidata.org'
  
  Wikidata Query Service (WDQS)
  
  - 'query-preview.wikidata.org'
  - 'query.wikidata.org'
  
  Errors
  
  - 'www.m.wikidata.org' (likely a faulty input)
  
  For this task we focus only on Wikidata's mobile and desktop interfaces.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-16 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices

2023-08-14 Thread Manuel

Manuel renamed this task from "[Analytics] Identify access via mobile devices 
independent of tags   " to "[Analytics] Identify access via mobile devices ".
Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-14 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread Manuel

Manuel added a comment.


  Ah, I see! This task was only about finding a way to reliably identify all 
uses (viewing and editing) from mobile devices, independent of their tags.
  
  Once we have a reliable way of detecting the user device type, the next goal 
will be along the lines of what you wrote, so it's very okay to invest in this.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  Or am I just jumping to the question in the description and we just want to 
figure out mobile edits and views over the period?

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  I guess I'm confused what the goal here is then 樂 As I understand it we're 
looking for users who are using the normal desktop UI on a mobile device. For 
the `wmf.webrequest` table we'd then use:
  
  - `user_agent` -> device used via `python-user-agent` and 
`user_agent.parse(ua_value).is_mobile`
  - `access_method` -> UI used
  
  We then get a breakdown that includes those who have a `desktop` 
`access_method` but are derived to be using a mobile device as well as the 
other combinations.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread Manuel

Manuel added a comment.


  Yes, I edited my comment, as I missed this important point: access_method 
just checks the URL for indications of a mobile UI, so it is about UI and not 
about device. So it is out of scope here, after all.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  My understanding of `access_method` is that it's only related to `user_agent` 
for mobile apps:
  
  > Mobile app requests are identified by the user agent including 
`WikipediaApp` or `Wikipedia/5.0`.
  
  For web requests it's based on the subdomains.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  I've already checked and there are differences between a `python-user-agents` 
derived device via `user_agents.parse(ua_value).is_mobile` and the 
`access_method`. Specifically we are getting users where the device from 
`.is_mobile` is `mobile`, but the access method is `desktop` implying that 
they're not using a `m.URL`.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread Manuel

Manuel added a comment.


  > Are we trying then the combination of user_agent_var.is_mobile = True and 
access_method = "desktop" via the access_method column from wmf.webrequest?
  
  It is important to keep user devices and Wikidata UI separate here. For now 
we are only trying to get the device side right.
  
  From your notes in the description I took that `access_method` is related to 
the UI shown and not the device used. But based on Wikitech 
 
and your comment T336361#9084157 
 it seems to only be a pre 
aggregation of user agents.
  
  So all of these would be alternative ways to get to the type of our users' 
devices:
  
  - `access_method` (pre-aggregation in `webrequests`)
  - the code linked above (used in the context of Varnish)
  - is_mobile (using python-user-agents)
  
  My guess would be that python-user-agents uses the most advanced algorithm of 
the three. On the other hand, `access_method` would be very convenient.
  
  So could you please quickly test how well `access_method` is doing in 
comparison? A week of distinct user agents would do the trick, I believe:
  
  - distinct user agents
  - is_mobile value
  - `access_method`
  - #of accesses
  
  The interesting parts would be where the two sources are not aligned.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread AndrewTavis_WMDE

AndrewTavis_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-11 Thread AndrewTavis_WMDE

AndrewTavis_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-10 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-10 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  @Manuel, I've been using python-user-agents 
 and so far it's going ok in so 
far as the `.ismobile` method seems to be working well. Are we trying then the 
combination of `user_agent_var.ismobile = True` and `access_method = "desktop"` 
via the `access_method` column from wmf.webrequest 
? 
For this column:
  
  > Mobile web requests are identified by the hostname containing a subdomain 
of `m`, `zero`, `wap`, or `mobile`. Any other request is classified as 
`desktop`.
  
  Let me know if this sounds alright to you :)

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-10 Thread AndrewTavis_WMDE

AndrewTavis_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-10 Thread AndrewTavis_WMDE

AndrewTavis_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-10 Thread AndrewTavis_WMDE

AndrewTavis_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-10 Thread AndrewTavis_WMDE

AndrewTavis_WMDE added a comment.


  @Manuel, just a note on using the `wmf.webrequest` table: now that I'm using 
Spark a bit more and can see the number of steps, it's definitely worth it to 
try to restrict the data based on the year and month as we're doing. Selecting 
30 days over two months takes dramatically longer than if we set the month 
column in the `WHERE` clause - roughly three times longer based on number of 
steps.
  
  Some bad news, the following query returns no distinct values for the 
`ch_ua_mobile` column:
  

SELECT
DISTINCT(ch_ua_mobile) AS ch_ua_mobile_values

FROM 
wmf.webrequest

WHERE
year = 2023
AND month = 7
AND '2023-07-01' <= dt
AND dt < '2023-08-01'
AND uri_host = 'www.wikidata.org'
  
  I've asked in `working-with-data` as we discussed :)

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-10 Thread AndrewTavis_WMDE

AndrewTavis_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-09 Thread AndrewTavis_WMDE

AndrewTavis_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-09 Thread AndrewTavis_WMDE

AndrewTavis_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-09 Thread Manuel

Manuel added a comment.


  That was very helpful, thank you, @Michael!

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-09 Thread Michael

Michael added a comment.


  > How do we deal with API consumers that disguise themselves as browsers?
  
  I'm not fully sure if that is true across the board, but from a quick look, 
it seems that API requests from the browser seem to usually have a referrer 
header set. Also, they usually have some cookies attached to them. I do not 
know if these pieces of information are available in the data you're looking 
at, and you might have better ones anyway. Just wanted share my quick thoughts 
from the perspective of dev that works on the interface.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Michael
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-09 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-09 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-09 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-09 Thread Manuel

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T336361: [Analytics] Identify access via mobile devices independent of tags

2023-08-09 Thread Manuel

Manuel renamed this task from "[Analytics] Identify access via mobile phones 
independent of tags   " to "[Analytics] Identify access via mobile devices 
independent of tags   ".
Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T336361

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndrewTavis_WMDE, Manuel
Cc: AndrewTavis_WMDE, Michael, Manuel, Aklapper, Danny_Benjafield_WMDE, 
Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

46 matches

Mail list logo