Ouch, i totally forgot the fixed width there, fixed tables below. On Wed, Feb 10, 2016 at 6:56 PM, Erik Bernhardson < [email protected]> wrote:
> Thanks for this list of possible metrics, lots of things to think about! > We don't yet collect enough data for all of these, but some of them we do. > A few potentially interesting tables: > > Number of searches per session: > +------------------+--------------+---------+ > | session_searches | num_sessions | percent | > +------------------+--------------+---------+ > | 1 | 59850 | 69.35% | > | 2 | 16564 | 19.19% | > | 3 | 5481 | 6.35% | > | 4 | 2176 | 2.52% | > | 5 | 1001 | 1.16% | > | 6 | 531 | 0.62% | > | 7 | 307 | 0.36% | > | 8 | 204 | 0.24% | > | 9 | 115 | 0.13% | > | 10 | 75 | 0.09% | > +------------------+--------------+---------+ > > Click through position by number of searches in session: > +----------------+-------+-------+-------+-------+-------+-------+ > | event_position | one | two | three | four | five | six | > +----------------+-------+-------+-------+-------+-------+-------+ > | 1 | 62.99 | 68.97 | 63.63 | 62.88 | 61.16 | 59.43 | > | 2 | 15.42 | 12.06 | 13.78 | 15.07 | 12.43 | 12.74 | > | 3 | 7.29 | 5.77 | 6.5 | 5.98 | 8.98 | 8.43 | > | 4 | 4.04 | 3.6 | 4.68 | 4.01 | 5.48 | 4.22 | > | 5 | 2.48 | 2.45 | 2.54 | 2.4 | 2.51 | 3.24 | > | 6 | 1.68 | 1.24 | 1.71 | 1.53 | 1.77 | 1.18 | > | 7 | 1.24 | 1.12 | 1.49 | 1.65 | 1.8 | 2.53 | > | 8 | 0.75 | 0.82 | 1.04 | 0.81 | 0.56 | 1.42 | > | 9 | 0.74 | 0.53 | 0.87 | 1.44 | 0.91 | 0.93 | > | 10 | 0.42 | 0.57 | 0.35 | 0.79 | 0.36 | 1.44 | > | 11 | 0.4 | 0.52 | 0.63 | 0.72 | 0.35 | 1.46 | > | 12 | 0.36 | 0.28 | 0.42 | 0.6 | 0.61 | 0.91 | > | 13 | 0.33 | 0.19 | 0.3 | 0.33 | 0.32 | 0.38 | > | 14 | 0.29 | 0.38 | 0.2 | 0.53 | 0.6 | 0.13 | > | 15 | 0.26 | 0.28 | 0.21 | 0.37 | 0.92 | 0.4 | > | 16 | 0.26 | 0.18 | 0.2 | 0.23 | 0.1 | 0 | > | 17 | 0.24 | 0.11 | 0.31 | 0.03 | 0.11 | 0.27 | > | 18 | 0.21 | 0.26 | 0.21 | 0.17 | 0.11 | 0.09 | > | 19 | 0.21 | 0.31 | 0.23 | 0.29 | 0.43 | 0.09 | > | 20 | 0.28 | 0.25 | 0.6 | 0.19 | 0.28 | 0.62 | > | 21 | 0.01 | 0.06 | 0.01 | 0 | 0.03 | 0 | > | 22 | 0.03 | 0.01 | 0.04 | 0 | 0.04 | 0.02 | > | 23 | 0.01 | 0.02 | 0 | 0 | 0.06 | 0 | > | 24 | 0.02 | 0.01 | 0.01 | 0 | 0.04 | 0.07 | > | 25 | 0.02 | 0.01 | 0.03 | 0 | 0.04 | 0 | > +----------------+-------+-------+-------+-------+-------+-------+ > > Click through position vs duration of search session in seconds. Column's > are named as min_max > > +----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+ > | event_position | 0_1 | 2_15 | 16_30 | 31_60 | 61_120 | 121_180 | > 181_240 | 241_300 | 301_600 | 601_1200 | 1201_1800 | > > +----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+ > | 1 | 92.52 | 72.07 | 58.59 | 58 | 56.79 | 57.1 | > 59.29 | 58.82 | 58.17 | 54.94 | 60.23 | > | 2 | 4.43 | 14.34 | 14.29 | 13.72 | 15.22 | 15.14 | > 13.2 | 16.3 | 15.08 | 14.8 | 11.13 | > | 3 | 1.62 | 5.83 | 8.6 | 7.85 | 7.1 | 8.17 | > 7.1 | 4.63 | 7.86 | 7.69 | 7.24 | > | 4 | 0.05 | 3.16 | 5.1 | 4.8 | 4.73 | 4.52 | > 5.61 | 4.43 | 4.75 | 4.83 | 4.79 | > | 5 | 0.5 | 1.59 | 3.7 | 3.5 | 3.81 | 2.27 | > 2.64 | 2.57 | 2.9 | 2.78 | 2.74 | > | 6 | 0.34 | 0.97 | 2.06 | 2.09 | 2.17 | 1.59 | > 2.15 | 3.48 | 1.81 | 2.57 | 0.86 | > | 7 | 0 | 0.67 | 1.41 | 1.84 | 1.63 | 2.26 | > 1.64 | 1.5 | 2.03 | 2.12 | 2.16 | > | 8 | 0 | 0.27 | 0.9 | 1.52 | 1.32 | 1.17 | > 1.39 | 1.1 | 1.28 | 1.46 | 1.28 | > | 9 | 0 | 0.23 | 1.15 | 1.16 | 1.04 | 1.02 | > 0.47 | 1.77 | 1.13 | 1.41 | 1.26 | > | 10 | 0 | 0.14 | 0.55 | 0.96 | 0.99 | 0.93 | > 0.76 | 0.75 | 0.45 | 1.17 | 0.86 | > | 11 | 0 | 0.17 | 0.41 | 0.76 | 0.66 | 0.64 | > 0.53 | 1.23 | 0.73 | 1.31 | 0.99 | > | 12 | 0.03 | 0.08 | 0.58 | 0.59 | 0.74 | 0.65 | > 0.55 | 0.19 | 0.45 | 0.92 | 1.32 | > | 13 | 0 | 0.08 | 0.49 | 0.39 | 0.67 | 0.4 | > 0.28 | 0.83 | 0.41 | 0.24 | 0.86 | > | 14 | 0 | 0.05 | 0.35 | 0.52 | 0.74 | 0.72 | > 0.57 | 0.51 | 0.37 | 0.77 | 0.86 | > | 15 | 0.16 | 0.05 | 0.41 | 0.41 | 0.37 | 0.35 | > 0.7 | 0.16 | 0.59 | 0.54 | 0.79 | > | 16 | 0.34 | 0.09 | 0.36 | 0.21 | 0.53 | 0.31 | > 0.35 | 0.35 | 0.23 | 0.43 | 0.46 | > | 17 | 0 | 0.07 | 0.17 | 0.32 | 0.18 | 0.56 | > 0.52 | 0.31 | 0.31 | 0.24 | 0.46 | > | 18 | 0 | 0.04 | 0.22 | 0.58 | 0.36 | 0.33 | > 0.67 | 0.28 | 0.25 | 0.53 | 0.75 | > | 19 | 0 | 0.05 | 0.2 | 0.29 | 0.42 | 0.81 | > 0.75 | 0.32 | 0.4 | 0.4 | 0.22 | > | 20 | 0 | 0.02 | 0.44 | 0.48 | 0.48 | 0.44 | > 0.74 | 0.36 | 0.64 | 0.53 | 0.31 | > | 21 | 0 | 0 | 0.01 | 0 | 0.03 | 0 | > 0 | 0.04 | 0.06 | 0.17 | 0.09 | > | 22 | 0 | 0.02 | 0 | 0 | 0 | 0.12 | > 0.01 | 0.01 | 0.05 | 0.03 | 0.24 | > | 23 | 0 | 0 | 0 | 0 | 0 | 0.03 | > 0.05 | 0 | 0.01 | 0.07 | 0.04 | > | 24 | 0 | 0 | 0 | 0 | 0 | 0.13 | > 0 | 0.03 | 0.03 | 0.03 | 0 | > | 25 | 0 | 0 | 0 | 0.02 | 0.01 | 0.33 | > 0 | 0.03 | 0.01 | 0.04 | 0.02 | > > +----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+ > > Number of searches performed vs number of clicks on search results per > session: > +----------+--------+--------+-------+-------+-------+ > | searches | zero | one | two | 3_5 | gt5 | > +----------+--------+--------+-------+-------+-------+ > | 0 | 0.00% | 0.08% | 0.00% | 0.00% | 0.00% | > | 1 | 40.85% | 19.77% | 0.27% | 0.05% | 0.01% | > | 2 | 10.56% | 7.86% | 1.46% | 0.06% | 0.00% | > | 3 | 4.75% | 2.55% | 0.98% | 0.27% | 0.00% | > | 4 | 1.91% | 1.25% | 0.51% | 0.29% | 0.00% | > | 5 | 1.14% | 0.55% | 0.30% | 0.23% | 0.01% | > | 6 | 0.61% | 0.31% | 0.17% | 0.18% | 0.02% | > | 7 | 0.37% | 0.19% | 0.11% | 0.16% | 0.02% | > | 8 | 0.24% | 0.10% | 0.08% | 0.10% | 0.02% | > | 9 | 0.14% | 0.06% | 0.06% | 0.07% | 0.02% | > | 10 | 0.11% | 0.06% | 0.03% | 0.04% | 0.02% | > | 11 | 0.07% | 0.03% | 0.01% | 0.04% | 0.03% | > | 12 | 0.06% | 0.02% | 0.01% | 0.02% | 0.02% | > | 13 | 0.03% | 0.01% | 0.01% | 0.02% | 0.01% | > | 14 | 0.03% | 0.01% | 0.02% | 0.02% | 0.01% | > +----------+--------+--------+-------+-------+-------+ > > This isn't nearly all the things you mentioned, but a few that we have > collected data for already. Hopefully these and a few other metrics can > become part of how we measure what effect changes to search make. > > > On Wed, Feb 10, 2016 at 3:01 AM, Justin Ormont <[email protected]> > wrote: > >> This is great. Do you have any categories tracked that could be >> interesting to break the position click-rates down by? eg: navigational vs. >> explorative queries, SAT clicks (satisfied user's query intent) vs. DSAT >> clicks (not satisfied), requery rate (how many times a user reformulates a >> new query in a session), time-to-first-click, search session duration, >> user's country/default language/# edits, length of query (# of query >> tokens), # of query results, popular vs. uncommon queries, high scoring >> SERP vs. low scoring SERP (or a proxy like Max BM25F of the top result), >> speller was click vs. not clicked, category of page clicked on, popular >> pages vs. rarely visited pages, etc. >> >> This experiment running on Special:Search is nice as that page doesn't >> automatically redirect when the query exactly matches a page. >> >> You can measure the positional importance by setting up an A/B test where >> you flip position 2 & 3. Also, a slowdown experiment would tell you the >> impact of latency, and help focus engineering efforts towards precision, or >> latency improvements. >> >> --justin >> >> >> On Tue, Feb 9, 2016 at 9:53 AM, Erik Bernhardson < >> [email protected]> wrote: >> >>> 2 | 34214 | 14.26% >> >> >> >> >> _______________________________________________ >> discovery mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/discovery >> >> >
_______________________________________________ discovery mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/discovery
