Hi All,
Earlier today we had our weekly FxA backend catchup, where we take some
time to sync up with :jrgm and :jbuck and discussion operational and
deployment issues. For those who missed it, these were the highlights:
* train-69 is being deployed this week, and you can read about the
operational changes in the (sorry, moco-confidential) google doc at [1].
The google doc improves on our previous google sheet for tracking ops
changes, due to a significant increase in the emoji density coefficient.
* :jrgm is starting the long, slow process of clearing expired OAuth
tokens from the database. There are many gigs of them and we estimate
it will take several days to clear them at a rate slow enough to avoid a
thundering herd of refresh attempts. But so far so good!
* We're seeing periodic spikes in read IOPS on the production database
which, after further investigation, appear to be caused by a nightly
metrics-gathering script that's doing a full table scan. Moving this
off to the read replica should eliminate the spurts of 500 errors that
have recently been occurring.
* Are you hapi? The auth-server is, since it's now running on node v4
and we can update it to the latest version of our web framework:
https://github.com/mozilla/fxa-auth-server/pull/1330
https://hapi.dev.lcip.org
* Once train-69 is out the door, we can try similar node v4 and hapi
updates for the oauth-server and profile-server.
* Vijay's work on the ip blocklist code is ready to land, and we had
some good discussion about the right way to keep these blocklists
updated in production. The details will be worked out in followups this
week.
* Our CSP reporting endpoint sometimes throws 500 errors, and we don't
really know why. It may be due to large or malformed payloads that are
being rejected at the nginx level. Anyone with a theory, please dive in
at https://github.com/mozilla/fxa-content-server/issues/4031
Cheers,
Ryan
[1]
https://docs.google.com/document/d/1lc5T1ZvQZlhXY6j1l_VMeQT9rs1mN7yYIcHbRPR2IbQ/
_______________________________________________
Dev-fxacct mailing list
[email protected]
https://mail.mozilla.org/listinfo/dev-fxacct
