Argggg badly named events.... 'save' is actually hitting the preview button (this is why the results vary so much between save and success). What I actually wanted was submit.
The same conclusion as above holds, but let's do those sql queries/maths again for completion: Pre A-B test, successful save events [1] 0 11872 1 13299 (25171 saves in both groups, 1427 more in group A - 5.7%) Post A/B test successful save events [2] 0 10075 1 10316 (20391 saves in both groups, 241 more in group A - 1.2%) New users pre A-B test successful save events [3] 0 5643 1 5829 (11472 saves in both groups, 186 more in group A - 1.6%) New users post A-B test successful save events [4] 0 4163 1 4359 (8522 saves in both groups, 196 more in group A - 2.3%) [A] https://gerrit.wikimedia.org/r/162164 [1] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'submit' and timestamp > 20140827000000 and timestamp < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_mobileMode = 'stable' group by event_isTestA [2] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'submit' and timestamp > 20140912000000 and timestamp < 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_mobileMode = 'stable' group by event_isTestA [3] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'submit' and timestamp > 20140827000000 and timestamp < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_userEditCount < 5 and event_mobileMode = 'stable' group by event_isTestA [4] select event_isTestA, count(*) from MobileWebEditing_8599025 where event_action = 'submit' and timestamp > 20140912000000 and timestamp< 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' and event_userEditCount < 5 and event_mobileMode = 'stable' group by event_isTestA On Mon, Sep 22, 2014 at 4:42 PM, Jon Robson <[email protected]> wrote: > Thanks for this link Nemo! > > I've discovered a caveat over the weekend which seems to make these > results more unreliable. > In test A, the success event is logged and then the page reloaded. > Concerned that the event might be lost in the page reload and skew the > results I decided to look at the 'save' event which is logged whenever > the user hits the save button. In comparison the 'success' event is > only logged when the save is successful. > > Pre A-B test, successful save events [1] > 0 14350 > 1 16518 > (30868 saves in both groups, 2168 more in group A, 7%) > > Post A/B test successful save events [2] > 0 13462 > 1 13273 > (26735 saves in both groups, 189 more in group B - 0.07%) > > New users pre A-B test successful save events [3] > 0 6216 > 1 6362 > (12578 saves in both groups, 146 more in group A - 1%) > > New users post A-B test successful save events [4] > 0 4870 > 1 4772 > (9642 saves in both groups, 98 more in group B - 1%) > > These show little variance amongst the groups and lead me to believe > that the save data I posted earlier cannot be relied on. > > I've written a patch to ensue the events get logged appropriately [A] > but I think from this my own conclusion is that ajax loading a page > via JavaScript after an edit only makes a significant difference (if > no difference at all) to the number of edits a user goes on to make. > We can probably safely kill it without concern. > > [A] https://gerrit.wikimedia.org/r/162164 > > [1] select event_isTestA, count(*) from MobileWebEditing_8599025 where > event_action = 'save' and timestamp > 20140827000000 and timestamp > < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' > and event_mobileMode = 'stable' group by event_isTestA > > [2] select event_isTestA, count(*) from MobileWebEditing_8599025 where > event_action = 'save' and timestamp > 20140912000000 and timestamp > < 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' > and event_mobileMode = 'stable' group by event_isTestA > > [3] select event_isTestA, count(*) from MobileWebEditing_8599025 where > event_action = 'save' and timestamp > 20140827000000 and timestamp > < 20140904000000 and event_editor = 'SourceEditor' and wiki ='enwiki' > and event_userEditCount < 5 and event_mobileMode = 'stable' group by > event_isTestA > > [4] select event_isTestA, count(*) from MobileWebEditing_8599025 where > event_action = 'save' and timestamp > 20140912000000 and timestamp > < 20140919000000 and event_editor = 'SourceEditor' and wiki ='enwiki' > and event_userEditCount < 5 and event_mobileMode = 'stable' group by > event_isTestA > > On Fri, Sep 19, 2014 at 11:27 PM, Federico Leva (Nemo) > <[email protected]> wrote: >> Thanks. For reference, the tracking bug for this area is: Bug 14123 – Make >> MediaWiki more AJAXy >> https://bugzilla.wikimedia.org/showdependencytree.cgi?id=14123&hide_resolved=1 >> Hoo *tripled* the number of patrollers in en.wiki by making patrol AJAXy >> (bug 7851/15936), so it's definitely an area worth working on. >> >> Nemo _______________________________________________ Mobile-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mobile-l
