On the apps, yep. On Thu, May 28, 2015 at 9:43 AM, Oliver Keyes <[email protected]> wrote:
> It wasn't previously on iOS, so we've just been tracking Android? :/ > > On 28 May 2015 at 12:34, Adam Baso <[email protected]> wrote: > > +Corey Floyd, who recently added code for this stuff on iOS (slated for > > forthcoming 4.1.4 iOS Wikipedia app release). > > > > On Wed, May 27, 2015 at 12:33 PM, Adam Baso <[email protected]> wrote: > >> > >> Thought I'd step in here. People who know the mechanics of the relevant > >> logging are as follows: > >> > >> Android: Dmitry Brant > >> Desktop: Bahodir (Baha) Mansurov > >> Mobile Web: Sam Smith & Baha > >> > >> CC'ing them. > >> > >> As I understand, Dan Garry's been talking with Baha already on the > desktop > >> piece. > >> > >> It looks like on Chrome/42 UAs the clickthrough rate for a given suggest > >> search form interaction is about 40%. [1] > >> > >> A couple of patches are pending (JS for emitting event on new EL schema) > >> that will make it possible to figure out how often form submission > within a > >> form interaction (ENTER/RETURN, tapping the magnifying class) occurs as > >> well. > >> > >> Total form interactions (keys on userSessionToken...maybe a better name > >> could be used) minus clickthroughs (click-result) minus form submission > >> (submit-form in the pending patches on the new EL schema) would be a > rough > >> proxy of abandonment, I think. > >> > >> I had heard sendBeacon capable UAs were likely to have greater success > >> emitting the click-result (i.e., when user clicks on a suggestion from > the > >> form on the search panel on desktop) event via mw.track, so it may make > >> sense to confine queries for such analysis on desktop to known > sendBeacon > >> browsers [2] to increase the odds of high fidelity data just in case > there > >> are outlier browsers that manage to somehow emit click-result events > through > >> means other than sendBeacon (it seems there may be some of these, > assuming > >> non-forged UAs). > >> > >> -Adam > >> > >> [1] > >> > >> > SELECT count(*) FROM Search_11670541 WHERE timestamp >= '20150526' AND > >> > timestamp < '20150527' AND event_action = 'click-result' AND wiki = > 'enwiki' > >> > and userAgent LIKE '%Chrome/42%'; > >> +----------+ > >> | count(*) | > >> +----------+ > >> | 112 | > >> +----------+ > >> 1 row in set (2.96 sec) > >> > >> > SELECT count(DISTINCT event_userSessionToken) FROM Search_11670541 > WHERE > >> > timestamp >= '20150526' AND timestamp < '20150527' AND event_action = > >> > 'click-result' AND wiki = 'enwiki' and userAgent LIKE '%Chrome/42%'; > >> +----------------------------------------+ > >> | count(DISTINCT event_userSessionToken) | > >> +----------------------------------------+ > >> | 112 | > >> +----------------------------------------+ > >> 1 row in set (38.06 sec) > >> > >> > >> > SELECT count(DISTINCT event_userSessionToken) FROM Search_11670541 > WHERE > >> > timestamp > '20150526' AND timestamp < '20150527' AND wiki = 'enwiki' > and > >> > userAgent LIKE '%Chrome/42%'; > >> +----------------------------------------+ > >> | count(DISTINCT event_userSessionToken) | > >> +----------------------------------------+ > >> | 286 | > >> +----------------------------------------+ > >> 1 row in set (7.26 sec) > >> > >> > >> [2] https://developer.mozilla.org/it/docs/Web/API/Navigator/sendBeacon > >> > >> On Tue, May 26, 2015 at 4:37 PM, Oliver Keyes <[email protected]> > >> wrote: > >>> > >>> Thanks Tomasz; great feedback! In order: > >>> > >>> * yeah, top percentiles were a heavily-requested thing so I built it > >>> in from the get-go. Similarly, mean/median so we have some ability to > >>> avoid distorting the results when the distribution changes. > >>> * The 3 days data thing is a known - > >>> https://phabricator.wikimedia.org/T100056 - and is next on my to-do > >>> list for bugfixes :). > >>> * Glad you like the interface! It's actually functional on mobile, too > >>> :D. > >>> * Sample rate is crucial, yep. I'm reaching out to the authors of the > >>> relevant EL schemas to find out how each was handled. > >>> * Sessions < results opened makes sense in the event that users didn't > >>> find what they wanted and went back to try again, but I'm not sure how > >>> "session" is calculated; this is again something we lack transparency > >>> around :(. Dan? You're the apps wizard. > >>> > >>> In supporting this: probably nothing at the moment although Nik/Kevin > >>> chipping in on the relevant phabricator ticket > >>> (https://phabricator.wikimedia.org/T99762 ) to validate how much of a > >>> PITA the idea of a unified schema and the associated implementations > >>> are, would be good. > >>> > >>> I'm sort of shocked to hear "we're supposed to be presenting this data > >>> at the next metrics meeting": in the future if there are instances > >>> where data is going to be up for public scrutiny, would it be possible > >>> to explicitly associate time for that? My goal is to get us to the > >>> point where our data is reliable all, or at least, most of the time, > >>> and for a fragment of one person's time over two weeks, I think > >>> progress on that is pretty fantastic. But prepping data for that kind > >>> of event does change the priorities and what tasks should be worked > >>> on. > >>> > >>> If we want to present data, generally speaking, let's discuss what we > >>> can show off. If we want to present the dashboards I'll put my all > >>> into making the data at least something where we know the > >>> deficiencies, if not something where we consider the deficiencies > >>> tolerable. > >>> > >>> On 26 May 2015 at 19:24, Tomasz Finc <[email protected]> wrote: > >>> > Thanks Oliver > >>> > > >>> > Early observations > >>> > > >>> > * Really happy to see top percentiles in load graphs > >>> > * Mobile Web has only three days data > >>> > * Interface is simple and easy to use > >>> > * We need to know the sample rate > >>> > * Apps have fewer sessions than results page opened > >>> > > >>> > Speaking over IRC it's clear that we don't have confidence in this > >>> > data. We need to fix this and fix it quickly so that we can > accurately > >>> > plan our work. We're supposed to be presenting this data at the next > >>> > metrics meeting and we're not a point where I feel comfortable > sharing > >>> > our data let alone next steps. > >>> > > >>> > Oliver & Dan, what can the team do to support you guys on this? I > want > >>> > you guys to own this and know that were here to support you. > >>> > > >>> > Should I be adding new feature requests and bugs to > >>> > https://phabricator.wikimedia.org/tag/search-data-analytics/ ? > >>> > > >>> > --tomasz > >>> > > >>> > On Tue, May 26, 2015 at 11:04 AM, James Douglas > >>> > <[email protected]> wrote: > >>> >> This is a very exciting preview of things to come. > >>> >> > >>> >> Where are the data coming from? Am I just confused, or does "6 > search > >>> >> sessions per day" seem low? > >>> >> > >>> >> On Fri, May 22, 2015 at 2:35 PM, Oliver Keyes <[email protected] > > > >>> >> wrote: > >>> >>> > >>> >>> http://searchdata.wmflabs.org/ - boop! This was my Friday. > Previously > >>> >>> we were playing around with them and testing what we needed with a > >>> >>> static snapshot; these dashboards will now update once a day with > new > >>> >>> information. > >>> >>> > >>> >>> It has turned up some bugs ("is the mobile schema just not > running?") > >>> >>> and there are more metrics to add. But for the time being, is > >>> >>> progress > >>> >>> :) > >>> >>> > >>> >>> -- > >>> >>> Oliver Keyes > >>> >>> Research Analyst > >>> >>> Wikimedia Foundation > >>> >>> > >>> >>> _______________________________________________ > >>> >>> Wikimedia-search-private mailing list > >>> >>> [email protected] > >>> >>> > https://lists.wikimedia.org/mailman/listinfo/wikimedia-search-private > >>> >> > >>> >> > >>> >> > >>> >> _______________________________________________ > >>> >> Wikimedia-search-private mailing list > >>> >> [email protected] > >>> >> > https://lists.wikimedia.org/mailman/listinfo/wikimedia-search-private > >>> >> > >>> > > >>> > _______________________________________________ > >>> > Wikimedia-search-private mailing list > >>> > [email protected] > >>> > > https://lists.wikimedia.org/mailman/listinfo/wikimedia-search-private > >>> > >>> > >>> > >>> -- > >>> Oliver Keyes > >>> Research Analyst > >>> Wikimedia Foundation > >>> > >>> _______________________________________________ > >>> Analytics mailing list > >>> [email protected] > >>> https://lists.wikimedia.org/mailman/listinfo/analytics > >> > >> > > > > > > _______________________________________________ > > Analytics mailing list > > [email protected] > > https://lists.wikimedia.org/mailman/listinfo/analytics > > > > > > -- > Oliver Keyes > Research Analyst > Wikimedia Foundation > > _______________________________________________ > Analytics mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/analytics >
_______________________________________________ Analytics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/analytics
