Using the infrastructure that was built into the content server, a user could only be part of 1 A/B test at a time, even if tests were sufficiently "independent". The mentioned change is to allow the user to be part of more than 1 test. If two tests have the possibility of interfering with each other, we still have the capability to ensure the user is a member of at most one of those tests, so we haven't lost any functionality either.
Shane On Mon, Jan 30, 2017 at 6:23 PM, Alex Davis <[email protected]> wrote: > The last minute train-79 activity to roll out "Connect another device" to >> 25% of people has the side effect of allowing us to roll out multiple >> independent A/B tests with minimal fuss. > > > Can you guys elaborate on this? I'd love to learn more :) > > -- > Alex Davis // Mountain View > Product Manager // FxA & Sync > (415) 769-9247 > IRC & Slack: adavis > > On Fri, Jan 27, 2017 at 3:09 PM, Shane Tomlinson <[email protected]> > wrote: > >> >> >> *Theme: Train-80 has begun* >> This was a very dense and technical meeting, so the notes are a bit >> longer than usual. >> >> Action items from last time: >> >> - Train-79 was tagged once, twice, then a third time on the content >> server. The last minute flurry of activity was to handle two things: >> >> - Bump the number of users who would see the new "connect another >> device" flow to 25%. >> >> - "Docker all the things" was a bit rocky. Circle would report >> multiple test runs as fully successful, while run against a branch, and >> then fail on master. jbuck and vlad wrangled the Docker build into >> shape. >> >> New >> >> - Ryan Feeley has lots of new DataDog graphs related to connect >> another device. Results continue to be positive, and now we have a better >> idea of ow many people see each particular screen. A surprising number of >> people seem to be verifying in a 2nd, disconnected, Firefox instance. >> These >> users appear to be highly motivated to sign in on that 2nd device. The >> numbers also reveal that most users sign up and verify on the same device >> (not too surprising), meaning the forthcoming "Send a Firefox Mobile >> install link via SMS" feature should make it significantly easier for >> these >> users to connect their mobile devices. >> >> - Phil, Vijay and I had a short pow-wow about the "Send SMS" feature. >> I'm stretched thin, and Vijay has offered to help for the next week. We >> figured out the major components we'll need, and which are self-contained >> enough that Vijay could handle in a short amount of time. Vijay is going >> to >> dive in on customs-server and fxa-js-client work, Phil is going to handle >> the auth-server, and I've successfully removed myself from active >> development for at least a week. >> >> - Phil has been diving into the content server to re-generate >> flow-id's and flow start times when a user signs out. I think with [1], he >> has a winner of a PR that ends up simplifying several pieces of internal >> state management. >> >> - Divya's work to show OAuth scopes in "Devices and Apps" is ~2/3 of >> the way done and should be ready to go with train-80. >> >> - Vlad has received feedback on his Web Sessions work and is going to >> update the "Sessions API" on the auth server to also return associated >> device info. This effort is to reduce the number of XHR calls that need to >> be made to show a list of connected devices, OAuth apps, and normal web >> sessions. >> >> - The last minute train-79 activity to roll out "Connect another >> device" to 25% of people has the side effect of allowing us to roll out >> multiple independent A/B tests with minimal fuss. >> >> - Phil is also working on "Functional Testing All The Things All The >> Time" - a recurring problem is a checkin to the auth/oauth/profile servers >> or js-client sometimes cause an unexpected test failure on the content >> server. It's always a drag when it happens. Phil thinks so too, so is >> experimenting with running a subset of content server functional tests on >> every checkin to the auth server. From what I saw in IRC, he has managed >> to >> get the test run to complete in under 10 minutes. >> >> - In Hawaii, we spent a few hours with Mark H discussing how we could >> get around [2]. There are several ways a user can have an active Sync >> session and have the FxA front-end forget about their session. This >> happens >> when the browser has the user's sessionToken but the user's FxA >> localStorage gets wiped. There are several ways this can happen - either a >> profile reset, or by modifying history settings, or an explicit "forget >> history", and even e10s plays a part (though this may have been resolved). >> When this happens, users who click "Manage Account" have to sign in again. >> It's less than awesome. The decided upon approach is do finally do the >> WebChannel handshake we've been discussing for nearly 2 years where when >> FxA loads, it'd request from the browser it's view of the FxA world. The >> browser would respond with a sessionToken, email, and user id. The FxA >> front end would use that as the canonical source of truth. I have 2 >> patches, one to the browser, one to FxA, to make this happen. We got stuck >> on how to approach private browsing mode, discussion in [3]. Since Hawaii, >> nothing has been done and we'd like to finish this before forgetting about >> it again. >> >> - Vlad has opened an auth-server PR [4] that should give us more >> insights into why the DB connections on fxa-dev boxes sometimes go >> completely haywire. >> >> - New auth-server loads tests are in the works using [5], when tests >> are run, results will be automatically sent to DataDog. This effort is to >> see how changes in the auth-server affect load and responsiveness. >> >> - The question came up in IRC the other day,"What would an ideal >> dynamic configuration system look like?" Able has proven itself valuable, >> but we are using it in ways it's not meant to be used and that has >> down-sides. We are asking ourselves what a better dynamic configuration >> system would look like, and leave Able to do what it's good at - bucketing >> users into experiments. >> >> Action items: >> >> - Vlad to tag content-server train-79 >> - stomlinson to bug "someone" about moving forward with [3]. >> - stomlinson to update fxa-ci w/ newest servers. >> - vlad to look at never-ending shrinkwrap problem in auth-server#1612 >> - rfeeley to start new feature doc for [6] - sending sign-in >> confirmation to users via sync tab. >> >> Shane >> >> [1] - https://github.com/mozilla/fxa-content-server/pull/4676 >> [2] - https://github.com/mozilla/fxa-content-server/issues/4252 >> [3] - https://bugzilla.mozilla.org/show_bug.cgi?id=1323853 >> [4] - https://github.com/mozilla/fxa-auth-server/pull/1627 >> [5] - https://artillery.io/ >> [6] - https://github.com/mozilla/fxa-content-server/issues/4667 >> >> _______________________________________________ >> Dev-fxacct mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/dev-fxacct >> >> >
_______________________________________________ Dev-fxacct mailing list [email protected] https://mail.mozilla.org/listinfo/dev-fxacct
