Thanks for this list of possible metrics, lots of things to think about! We don't yet collect enough data for all of these, but some of them we do. A few potentially interesting tables:
Number of searches per session: +------------------+--------------+---------+ | session_searches | num_sessions | percent | +------------------+--------------+---------+ | 1 | 59850 | 69.35% | | 2 | 16564 | 19.19% | | 3 | 5481 | 6.35% | | 4 | 2176 | 2.52% | | 5 | 1001 | 1.16% | | 6 | 531 | 0.62% | | 7 | 307 | 0.36% | | 8 | 204 | 0.24% | | 9 | 115 | 0.13% | | 10 | 75 | 0.09% | +------------------+--------------+---------+ Click through position by number of searches in session: +----------------+-------+-------+-------+-------+-------+-------+ | event_position | one | two | three | four | five | six | +----------------+-------+-------+-------+-------+-------+-------+ | 1 | 62.99 | 68.97 | 63.63 | 62.88 | 61.16 | 59.43 | | 2 | 15.42 | 12.06 | 13.78 | 15.07 | 12.43 | 12.74 | | 3 | 7.29 | 5.77 | 6.5 | 5.98 | 8.98 | 8.43 | | 4 | 4.04 | 3.6 | 4.68 | 4.01 | 5.48 | 4.22 | | 5 | 2.48 | 2.45 | 2.54 | 2.4 | 2.51 | 3.24 | | 6 | 1.68 | 1.24 | 1.71 | 1.53 | 1.77 | 1.18 | | 7 | 1.24 | 1.12 | 1.49 | 1.65 | 1.8 | 2.53 | | 8 | 0.75 | 0.82 | 1.04 | 0.81 | 0.56 | 1.42 | | 9 | 0.74 | 0.53 | 0.87 | 1.44 | 0.91 | 0.93 | | 10 | 0.42 | 0.57 | 0.35 | 0.79 | 0.36 | 1.44 | | 11 | 0.4 | 0.52 | 0.63 | 0.72 | 0.35 | 1.46 | | 12 | 0.36 | 0.28 | 0.42 | 0.6 | 0.61 | 0.91 | | 13 | 0.33 | 0.19 | 0.3 | 0.33 | 0.32 | 0.38 | | 14 | 0.29 | 0.38 | 0.2 | 0.53 | 0.6 | 0.13 | | 15 | 0.26 | 0.28 | 0.21 | 0.37 | 0.92 | 0.4 | | 16 | 0.26 | 0.18 | 0.2 | 0.23 | 0.1 | 0 | | 17 | 0.24 | 0.11 | 0.31 | 0.03 | 0.11 | 0.27 | | 18 | 0.21 | 0.26 | 0.21 | 0.17 | 0.11 | 0.09 | | 19 | 0.21 | 0.31 | 0.23 | 0.29 | 0.43 | 0.09 | | 20 | 0.28 | 0.25 | 0.6 | 0.19 | 0.28 | 0.62 | | 21 | 0.01 | 0.06 | 0.01 | 0 | 0.03 | 0 | | 22 | 0.03 | 0.01 | 0.04 | 0 | 0.04 | 0.02 | | 23 | 0.01 | 0.02 | 0 | 0 | 0.06 | 0 | | 24 | 0.02 | 0.01 | 0.01 | 0 | 0.04 | 0.07 | | 25 | 0.02 | 0.01 | 0.03 | 0 | 0.04 | 0 | +----------------+-------+-------+-------+-------+-------+-------+ Click through position vs duration of search session in seconds. Column's are named as min_max +----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+ | event_position | 0_1 | 2_15 | 16_30 | 31_60 | 61_120 | 121_180 | 181_240 | 241_300 | 301_600 | 601_1200 | 1201_1800 | +----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+ | 1 | 92.52 | 72.07 | 58.59 | 58 | 56.79 | 57.1 | 59.29 | 58.82 | 58.17 | 54.94 | 60.23 | | 2 | 4.43 | 14.34 | 14.29 | 13.72 | 15.22 | 15.14 | 13.2 | 16.3 | 15.08 | 14.8 | 11.13 | | 3 | 1.62 | 5.83 | 8.6 | 7.85 | 7.1 | 8.17 | 7.1 | 4.63 | 7.86 | 7.69 | 7.24 | | 4 | 0.05 | 3.16 | 5.1 | 4.8 | 4.73 | 4.52 | 5.61 | 4.43 | 4.75 | 4.83 | 4.79 | | 5 | 0.5 | 1.59 | 3.7 | 3.5 | 3.81 | 2.27 | 2.64 | 2.57 | 2.9 | 2.78 | 2.74 | | 6 | 0.34 | 0.97 | 2.06 | 2.09 | 2.17 | 1.59 | 2.15 | 3.48 | 1.81 | 2.57 | 0.86 | | 7 | 0 | 0.67 | 1.41 | 1.84 | 1.63 | 2.26 | 1.64 | 1.5 | 2.03 | 2.12 | 2.16 | | 8 | 0 | 0.27 | 0.9 | 1.52 | 1.32 | 1.17 | 1.39 | 1.1 | 1.28 | 1.46 | 1.28 | | 9 | 0 | 0.23 | 1.15 | 1.16 | 1.04 | 1.02 | 0.47 | 1.77 | 1.13 | 1.41 | 1.26 | | 10 | 0 | 0.14 | 0.55 | 0.96 | 0.99 | 0.93 | 0.76 | 0.75 | 0.45 | 1.17 | 0.86 | | 11 | 0 | 0.17 | 0.41 | 0.76 | 0.66 | 0.64 | 0.53 | 1.23 | 0.73 | 1.31 | 0.99 | | 12 | 0.03 | 0.08 | 0.58 | 0.59 | 0.74 | 0.65 | 0.55 | 0.19 | 0.45 | 0.92 | 1.32 | | 13 | 0 | 0.08 | 0.49 | 0.39 | 0.67 | 0.4 | 0.28 | 0.83 | 0.41 | 0.24 | 0.86 | | 14 | 0 | 0.05 | 0.35 | 0.52 | 0.74 | 0.72 | 0.57 | 0.51 | 0.37 | 0.77 | 0.86 | | 15 | 0.16 | 0.05 | 0.41 | 0.41 | 0.37 | 0.35 | 0.7 | 0.16 | 0.59 | 0.54 | 0.79 | | 16 | 0.34 | 0.09 | 0.36 | 0.21 | 0.53 | 0.31 | 0.35 | 0.35 | 0.23 | 0.43 | 0.46 | | 17 | 0 | 0.07 | 0.17 | 0.32 | 0.18 | 0.56 | 0.52 | 0.31 | 0.31 | 0.24 | 0.46 | | 18 | 0 | 0.04 | 0.22 | 0.58 | 0.36 | 0.33 | 0.67 | 0.28 | 0.25 | 0.53 | 0.75 | | 19 | 0 | 0.05 | 0.2 | 0.29 | 0.42 | 0.81 | 0.75 | 0.32 | 0.4 | 0.4 | 0.22 | | 20 | 0 | 0.02 | 0.44 | 0.48 | 0.48 | 0.44 | 0.74 | 0.36 | 0.64 | 0.53 | 0.31 | | 21 | 0 | 0 | 0.01 | 0 | 0.03 | 0 | 0 | 0.04 | 0.06 | 0.17 | 0.09 | | 22 | 0 | 0.02 | 0 | 0 | 0 | 0.12 | 0.01 | 0.01 | 0.05 | 0.03 | 0.24 | | 23 | 0 | 0 | 0 | 0 | 0 | 0.03 | 0.05 | 0 | 0.01 | 0.07 | 0.04 | | 24 | 0 | 0 | 0 | 0 | 0 | 0.13 | 0 | 0.03 | 0.03 | 0.03 | 0 | | 25 | 0 | 0 | 0 | 0.02 | 0.01 | 0.33 | 0 | 0.03 | 0.01 | 0.04 | 0.02 | +----------------+-------+-------+-------+-------+--------+---------+---------+---------+---------+----------+-----------+ Number of searches performed vs number of clicks on search results per session: +----------+--------+--------+-------+-------+-------+ | searches | zero | one | two | 3_5 | gt5 | +----------+--------+--------+-------+-------+-------+ | 0 | 0.00% | 0.08% | 0.00% | 0.00% | 0.00% | | 1 | 40.85% | 19.77% | 0.27% | 0.05% | 0.01% | | 2 | 10.56% | 7.86% | 1.46% | 0.06% | 0.00% | | 3 | 4.75% | 2.55% | 0.98% | 0.27% | 0.00% | | 4 | 1.91% | 1.25% | 0.51% | 0.29% | 0.00% | | 5 | 1.14% | 0.55% | 0.30% | 0.23% | 0.01% | | 6 | 0.61% | 0.31% | 0.17% | 0.18% | 0.02% | | 7 | 0.37% | 0.19% | 0.11% | 0.16% | 0.02% | | 8 | 0.24% | 0.10% | 0.08% | 0.10% | 0.02% | | 9 | 0.14% | 0.06% | 0.06% | 0.07% | 0.02% | | 10 | 0.11% | 0.06% | 0.03% | 0.04% | 0.02% | | 11 | 0.07% | 0.03% | 0.01% | 0.04% | 0.03% | | 12 | 0.06% | 0.02% | 0.01% | 0.02% | 0.02% | | 13 | 0.03% | 0.01% | 0.01% | 0.02% | 0.01% | | 14 | 0.03% | 0.01% | 0.02% | 0.02% | 0.01% | +----------+--------+--------+-------+-------+-------+ This isn't nearly all the things you mentioned, but a few that we have collected data for already. Hopefully these and a few other metrics can become part of how we measure what effect changes to search make. On Wed, Feb 10, 2016 at 3:01 AM, Justin Ormont <[email protected]> wrote: > This is great. Do you have any categories tracked that could be > interesting to break the position click-rates down by? eg: navigational vs. > explorative queries, SAT clicks (satisfied user's query intent) vs. DSAT > clicks (not satisfied), requery rate (how many times a user reformulates a > new query in a session), time-to-first-click, search session duration, > user's country/default language/# edits, length of query (# of query > tokens), # of query results, popular vs. uncommon queries, high scoring > SERP vs. low scoring SERP (or a proxy like Max BM25F of the top result), > speller was click vs. not clicked, category of page clicked on, popular > pages vs. rarely visited pages, etc. > > This experiment running on Special:Search is nice as that page doesn't > automatically redirect when the query exactly matches a page. > > You can measure the positional importance by setting up an A/B test where > you flip position 2 & 3. Also, a slowdown experiment would tell you the > impact of latency, and help focus engineering efforts towards precision, or > latency improvements. > > --justin > > > On Tue, Feb 9, 2016 at 9:53 AM, Erik Bernhardson < > [email protected]> wrote: > >> 2 | 34214 | 14.26% > > > > > _______________________________________________ > discovery mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/discovery > >
_______________________________________________ discovery mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/discovery
