Re: rate-limiting for the channelserver

[Phil Booth]

Fwiw, I have two somewhat conflicting thoughts related to this:

1. The email service needs to grow rate-limiting soon and if there's a
standalone service available to use already, that will probably save me
some time.

2. Historically I've found the customs server code hard to grok. Part of me
dreads the prospect of working with it again.

Of course, extracting it into its own service might also involve improving
readability/maintainability, which would abate my dread. But if a 3rd-party
option can do the same job (I haven't dug into the ratelimit/limitd links
yet), I might be more inclined to take that option.

On Thu, Sep 20, 2018 at 6:11 AM, Julien Vehent  wrote:

> +alm & g-k
>
> On Thu, Sep 20, 2018, 00:52 Ryan Kelly  wrote:
>
>> On Thu, 20 Sep 2018 at 14:35, Ryan Kelly  wrote:
>>
>>>
>>> Hi All,
>>>
>>> Over in github we've been discussing our options of rate-limiting
>>> pairing channel creation attempts:
>>>
>>>   https://github.com/mozilla-services/channelserver/issues/21
>>>
>>> One obvious approach would be to use the existing fxa-customs-server,
>>> and just add some new action types like "createPairingChannel" and
>>> "connetToPairingChannel" that the channelserver can send over for
>>> checking.  However, the fxa-customs-server is currently run as a private
>>> "sidecar" service for fxa-auth-server, exposed only over a localhost
>>> interface.
>>>
>>> Does it make sense for us to try to extract fxa-customs-server into its
>>> own standalone service that can be accessed by multiple consumers?  Or is
>>> that likely to be more work than just adding rate-limiting code directly
>>> into the channelserver?
>>>
>>
>> Another option would be to try running a third-party ratelimiting daemon
>> that can be shared among different services, such as:
>>
>>   https://github.com/lyft/ratelimit
>>   https://github.com/limitd/limitd
>>
>> Which may be less work than adding custom rate-limiting code in
>> channelserver.
>>
>> +ulfr for possible opinions from opsec team.
>>
>>   Cheers,
>>
>> Ryan
>>
>
> ___
> Dev-fxacct mailing list
> Dev-fxacct@mozilla.org
> https://mail.mozilla.org/listinfo/dev-fxacct
>
>
___
Dev-fxacct mailing list
Dev-fxacct@mozilla.org
https://mail.mozilla.org/listinfo/dev-fxacct

FxA front-end meeting, 23-Feb-2017

[Phil Booth]

We just had the front-end catch-up. It was a tiny little tiddler of a
meeting today, these are the notes:

# Action items from last time

vlad to deploy web sessions to box
not done

shane to move sms feature doc to github
not done

shane to ask if we can release the sms feature in canada
not done

vlad to add `from` header in the auth mailer
done, train 80.2

train 81 cut
not done yet

shane to review greg's csp 404 patch
done

vlad to file issue about customs logging weirdness
not needed any more

# SMS install

work so far is all merged
phil working on an extra auth server endpoint
shane and rfeeley talked yesterday, updates are already in progress
resend email and back button coming on train 82
shane to write a qa doc for softvision

# Add CSP to 404 page

https://github.com/mozilla/fxa-content-server/pull/4750
shane reviewed greg's patch
uses (someone?)'s idea to examine the content-type header
ready for review

# Validating csp reports

https://github.com/mozilla/fxa-content-server/pull/4746
updated based on vlad's and sean's feedback
ready for review
for train 82

# Web sessions

private pr depends on public one
shane reviewing

# Recovery email notes

notes by alex, responses by rfk:
https://docs.google.com/document/d/1xki49BSoRgWPQxTH7eHhaUoWsYMthQBs8aIn-7b2YV4/

# Amplitude

vlad emailed everyone login details
has anybody tried it out yet? nope
it shows some weird results
funnel is upside down, everybody is completed, 20% have signed up
everyone to check vlad's email and give it a spin

# Mailcheck

finally deployed, after 2 years of development
all finished, thanks to divya

# Disabling statsd on the auth server

https://github.com/mozilla/fxa-auth-server/pull/1673
pr from vlad
needs review if anyone has time
those metrics are essentially useless

# Move email verification to server

in progress
https://github.com/mozilla/fxa-content-server/issues/4482
divya hoping to get it finished before she leaves
if it works out well we should look at moving sign in confirmation too

# Auth server patch 81.1

vijay is fixing "skip sign in confirmation if its a new account"
caused by a discrepancy between memdb and the stored procedure
not caught by tests
api docs say createdAt should be returned
vlad says no need to update shrinkwrap
vijay to update the deploy docs, it requires a db migration
___
Dev-fxacct mailing list
Dev-fxacct@mozilla.org
https://mail.mozilla.org/listinfo/dev-fxacct

Re: meeting summary: FxA Demo 2017-1-30

[Phil Booth]

Just to clarify one tiny thing in here:

On Tue, Jan 31, 2017 at 1:11 PM, Vijay Budhram  wrote:

> * pb followed up on a demo about fxa-test [4]. He created a custom
> organization for this testing. If you would like access, ask pb. The
> repository has the potential to be noisy with emails.
>

I don't think it has the potential to be noisy now. I thought it did when I
first set it up, because I wasn't sure how solid the builds were. They seem
solid.

I invited the members of @mozilla/fxa-devs to the GitHub org last night.
Depending on your CircleCI settings, it should let you know if you break
the build. This is good noise.
___
Dev-fxacct mailing list
Dev-fxacct@mozilla.org
https://mail.mozilla.org/listinfo/dev-fxacct

Re: US holiday on Monday - reschedule meetings until Tuesday?

[Phil Booth]

Works for me, although the 2nd half of the morning meeting would clash with
the monthly internal meeting in that case (but I don't mind watching the
recording).

On Fri, Jan 13, 2017 at 9:39 AM, Shane Tomlinson 
wrote:

> Hey all, you probably all saw Potch's email about it being MLK Jr Day in
> the US, and a third of our team will be absent from the meetings. I propose
> we reschedule all meetings to Tuesday, same time.
>
> Shane
>
> ___
> Dev-fxacct mailing list
> Dev-fxacct@mozilla.org
> https://mail.mozilla.org/listinfo/dev-fxacct
>
>
___
Dev-fxacct mailing list
Dev-fxacct@mozilla.org
https://mail.mozilla.org/listinfo/dev-fxacct

FxA meeting summary: backend meeting, 15-Nov-2016

[Phil Booth]

Hey everyone,

We just had our regular backend catch-up. It was not recorded but I did
take some stream-of-consciousness notes on what we discussed:

* train 74 was cut on time!

* but the deploy doc was empty!
  * a db migration was not mentioned
  * some config changes weren't mentioned
  * we should all try to remember to keep the deploy doc up to data as we go

* oauth token purging
  * on the back burner at the moment, last worked on a few weeks ago
  * currently up to november 2015, which contains twice as many tokens for
some reason
  * were 500s returned on requests from profile server?

* verification reminders disabled
  * no smoking gun except replication couldn't keep up
  * replication thread spending most time in system-locked state
  * there was a transaction that had millions of rows locked
* stored procedures make it difficult to see exactly what the problem is
* once that transaction unwinds, everything returns to a good state
  * it's recommended not to use temporary tables with replication, is that
the issue?
  * or could it be something to do with select for update?
  * do we need a consensus protocol for competing processes?
  * we don't actually need verification reminders running on all instances
anyway
* only needs to be 1 query every 30 seconds
* with replication we're making a query every second
* no need to compete
  * what are the release mechanics if we limit it to 1 instance?

* teamcity permissions
  * deployed a box to try out teamcity 10 with github integration
  * it wants write permissions to the github org, including private repos
  * means the teamcity box would need serious hardening
  * what if there's a zero-day exploit in teamcity?
  * vlad's going to email teamcity

* sentry
  * deployed with train 73 content server
  * where are all the errors?
* only two errors reported, one from git clone, one from git fetch
* "/experiments already exists and is not empty"
  * if things go wrong it should show up there

* profile server
  * docker stack is slow, seeing 4-10 times higher latency than non-docker
* seconds of latency
  * cpu is only a little bit higher
  * also docker stack never returns a 304 for /v1/profile
* usually there are around 4% or 5% 304s
* absolute dead zero inside docker
* how the heck can docker affect this?
* etag-related?
* maybe a significant clue of where the slowness comes from
  * try disabling all logging so there is no disk i/o, will that fix the
latency?
* log data is handled differently in docker
* different centos version, different log flow, uses systemd and
journald
  * one of the devs to try running in docker locally to debug the 304 issue

* profile server avatars
  * only one avatar per user
  * drop some tables
  * content server should have no more code for get-avatars api
  * if the endpoint is not being elsewhere (it shouldn't be), we should get
rid of it

And that was it.

Cheerio,
Pb
___
Dev-fxacct mailing list
Dev-fxacct@mozilla.org
https://mail.mozilla.org/listinfo/dev-fxacct

Meeting summary: product meeting, 27-Oct-2016

[Phil Booth]

Hey all, we just had our regular product meeting for Firefox Accounts.

Somebody else will reply to this in due course with a link to the
recording. In the meantime, here are the notes:

* Toronto work week
   * The plan was to try and brainstorm what to work on ahead of time, but
some key people weren't in the meeting so that's been postponed until the
Monday morning.
   * Perhaps Monday morning can include hanging wireframes on the wall and
picking the ones that people like best.
   * Alex suggested that it might make sense to avoid the lowest-hanging
fruit, as it's so rare to have people in the same room.
   * Important to think about what things might lead to, is something a
step on the way to some bigger feature we want down the line?
   * Vlad's not a massive fan of the login-via-QR-code feature, wonders if
something similar to the unblock code could be more effective.

* FxA-105: IP blocklist
  * Not deployed yet, on train 72.
  * Stays in the `measuring` column.

* FxA-89: Devices view (phase 1)
  * Rolled out to 60% of users.
  * Results summarised in the features doc:
 *
https://github.com/mozilla/fxa/blob/master/features/FxA-89-devices-view/README.md#phase-1-results
 * 38% of users click `Refresh`
 * 20% of users click `Disconnect`
* Of which, 65% actually confirm the disconnection.
 * Reasons for disconnection:
* 60% say old device.
* 30% would rather not say.
* 6% say the device is suspicious.
* 4% say the lost the device.
   * Will be rolled out to 100% of users in train 73.
   * Moved to the `done` column.

* FxA-106: Sign-in unblock
   * Not deployed yet, on train 72.
   * Some disagreement whether to roll out at 50% or 100%, think the latest
consensus was 50% in order to measure the effect on sign-ins.
   * There is some doubt about our ability to properly measure this.
   * Moved to the `measuring` column.

* FxA-57: Verification reminders
   * Punted to train 73, ran out of time this train.
   * Stays in the `shipping` column.

* FxA-108: Update all the deps
   * Finished with the update to Hapi and Joi in OAuth server.
   * Moved to the `measuring` column.

* FxA-70: KPI dashboards
   * Import of segmented and historical data is finished.
   * Just need to add segmentation to the graphs.
   * Stays in the `building` column.

* FxA-41: Sign-in funnel
   * Fixed a lot of stuff related to events emitted from the content and
auth servers.
   * There is a problem with timestamps in the data, some days are missing,
and we have some data from the future.
   * Alex and Phil to chat in Vidyo next week about mapping out all the
different possible flows, to understand what the funnel query should look
like.
   * Stays in the `building` column.

* Devices view (phase 2)
   * New icons have shipped.
   * Vlad to sync up with the two Ryan's to work out the rest of it.
   * Stays in the `building` column.

* FxA-15: Connected apps in the device view
   * Vlad to sync up with Ryan F.
   * Stays in the `building` column.

* FxA-107: Login event history (IP profiling)
   * Got pulled because the impact was not likely to be as high as first
thought.
   * Need to analyse server logs for likely impact.
   * Before committing to things in the future, we need to properly
validate the idea.
   * Should have been flagged during planning, then postponed until
validated.
   * Stays in the `designing` column.

* Improve end-to-end testing
   * New feature added by Vlad
   * QA is working on it, Vlad is helping them out
   * Moved to the `designing` column.

Cheerio!
___
Dev-fxacct mailing list
Dev-fxacct@mozilla.org
https://mail.mozilla.org/listinfo/dev-fxacct

meeting summary: front-end meeting, 13-Oct-2016

[Phil Booth]

Hey all, we just had our regular Thursday FxA front-end meeting.

I don't have a link to the recording or know where to get it from, but I'm
confident that one of my excellent compadres will reply to this email with
the link on my behalf.

I do have notes though. These are they:

* Toronto work week:
   * the focus is on increasing our number of multi-device users
   * accommodation has been booked
   * there should be an iOS engineer available to work with the team
   * lunch arrangements not yet made

* content server node 4 memory growth:
   * Shane has been trying to reproduce by repeatedly sending invalid
metrics data using gobench:
  * 500 concurrent clients for an hour at a time
  * memory usage grows and latency increases, then GC kicks in to clear
up and it never goes above 260 mb after that
   * jrgm has seen a difference in behaviour between the 70.2 and 70.3
releases:
  * 70.2 consistently shows the problem, 70.3 doesn't
  * the difference is not service-impacting, milliseconds in latency
  * this is the only code change:
 * https://github.com/mozilla/fxa-content-server/commit/
768addeda8c5cc83bcfdd409658d8d9c223b2198
  * train 71 is imminent, maybe magick fixed it

* train 72 strings will be cut tomorrow:
   * Shane has the only changes
   * Shane to cut strings
   * train 72 will be cut early next week

* lots of content server tests are not being run in travis:
   * Fx 40 barfs on `let`, those tests are silently dropped then the other
tests pass
   * Shane tried to update to Fx 46, that made all(?) the tests fail
   * Fx 47+ known not to work with Selenium WebDriver
   * Vijay is happily working with Fx 45 and Selenium 2.51
   * Shane uses Fx 46 and Selenium 2.53 locally
   * we need to make travis fail if Fx bails out
   * can we look at babelizing the tests?

* fxadeploybox needs to be updated for sign-in unblock
   * does only Vlad have permission to do this?
   * or is the box borked?
  * jrgm authenticated with ssh -vvv but then got logged out
immediately, maybe a sign that the disk is full
   * Shane proposes that we just set up a new box

* sign-in unblock
   * only 2 PRs left, 1 content server, 1 fxa-dev
   * we need all the tests to be fixed first, hence Shane addressing those
at the moment

* fxa-dev boxes can run node 6 now

* teamcity runs tests in node 6 now

* yarn is the new npm hotness
   * content server install with a cold cache takes 59 seconds
   * content server install with a warm cache takes 20 seconds
   * the post-yarn functional tests don't work yet
   * jrgm broke yarn when he tried it with the auth server

* there was something about a CSP bug and a fix to do with unicode
characters?
   * I was typing and not paying attention

Cheerio!
___
Dev-fxacct mailing list
Dev-fxacct@mozilla.org
https://mail.mozilla.org/listinfo/dev-fxacct

Re: Managing format of push payloads

[Phil Booth]

My $0.02: I'm pro-validation but ambivalent about JSON Schema specifically.
However it should be straightforward for the auth server to convert from
JSON Schema to Joi validation objects via, e.g.
https://www.npmjs.com/package/enjoi, provided we can stick to the subset
that is implemented there (or wherever).

On Thu, Apr 28, 2016 at 8:13 AM, Mark Hammond  wrote:

> Hi all,
>   Edouard has been working on adding payloads to our push messages and we
> were having a bit of a discussion about how to define these payloads. This
> is becoming relevant to FxA and to Sync, hence I'm copying both lists.
>
> When defining the data in a payload for a certain message, both the server
> and the client must agree on the format. Initially we can manage this in an
> ad-hoc basis, but I'm not sure this scales particularly well.
>
> Somewhat inspired by the new telemetry data pipeline, I'm wondering if we
> should consider moving towards using JSONSchema (json-schema.org) to
> define our payloads? The benefits I see here are:
>
> * A single format that the client and server can use to describe and agree
> on the data being exchanged. We'd need to agree on a canonical location for
> a single schema that the client and server agree on, but that's probably
> doable.
>
> * Eventually unit tests on both the client and server could use this
> schema to validate the messages they send (or expect to receive). This
> would also be useful when mocking - eg, unit tests in the browser could
> confirm that the mocked data (ie, data we pretend came from the server)
> also conforms to the schema, to help validate that we are indeed testing
> data that looks like what the server actually sends.
>
> * All data can be validated without knowing the contents of the message.
>
> * All messages are likely to be somewhat consistent, given someone will
> need to define the schema and will use as reference other existing schemas.
>
> This is somewhat aspirational (ie, I'm not suggesting we would do all that
> work up-front), but there's already a bug on file to perform client-side
> schema validations for telemetry (bug 1249925) and we could possibly
> piggy-back off that work for the client side.
>
> Does anyone think this is worthwhile, or am I over-thinking the problem at
> hand and ad-hoc management of the payload definitions is OK?
>
> Cheers,
>
> Mark
> ___
> Dev-fxacct mailing list
> Dev-fxacct@mozilla.org
> https://mail.mozilla.org/listinfo/dev-fxacct
>
___
Dev-fxacct mailing list
Dev-fxacct@mozilla.org
https://mail.mozilla.org/listinfo/dev-fxacct