From where would you propose measuring these data points? Obviously network latency will have a great impact on some of the metrics and a consistent location would help to define the pass/fail of each test. I do think another benchmark Ops "features" would be a set of latency-to-datacenter values, but I know that is a much harder taks. Thanks for putting this together.
On Thu, Mar 21, 2013 at 6:40 PM, Asher Feldman <afeld...@wikimedia.org>wrote: > I'd like to push for a codified set of minimum performance standards that > new mediawiki features must meet before they can be deployed to larger > wikimedia sites such as English Wikipedia, or be considered complete. > > These would look like (numbers pulled out of a hat, not actual > suggestions): > > - p999 (long tail) full page request latency of 2000ms > - p99 page request latency of 800ms > - p90 page request latency of 150ms > - p99 banner request latency of 80ms > - p90 banner request latency of 40ms > - p99 db query latency of 250ms > - p90 db query latency of 50ms > - 1000 write requests/sec (if applicable; writes operations must be free > from concurrency issues) > - guidelines about degrading gracefully > - specific limits on total resource consumption across the stack per > request > - etc.. > > Right now, varying amounts of effort are made to highlight potential > performance bottlenecks in code review, and engineers are encouraged to > profile and optimize their own code. But beyond "is the site still up for > everyone / are users complaining on the village pump / am I ranting in > irc", we've offered no guidelines as to what sort of request latency is > reasonable or acceptable. If a new feature (like aftv5, or flow) turns out > not to meet perf standards after deployment, that would be a high priority > bug and the feature may be disabled depending on the impact, or if not > addressed in a reasonable time frame. Obviously standards like this can't > be applied to certain existing parts of mediawiki, but systems other than > the parser or preprocessor that don't meet new standards should at least be > prioritized for improvement. > > Thoughts? > > Asher > _______________________________________________ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
