Diff
Modified: trunk/Websites/perf.webkit.org/ChangeLog (183231 => 183232)
--- trunk/Websites/perf.webkit.org/ChangeLog 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/ChangeLog 2015-04-24 01:16:37 UTC (rev 183232)
@@ -1,3 +1,97 @@
+2015-04-23 Ryosuke Niwa <[email protected]>
+
+ Perf dashboard should automatically detect regressions
+ https://bugs.webkit.org/show_bug.cgi?id=141443
+
+ Reviewed by Anders Carlsson.
+
+ Added a node.js script detect-changes.js to detect potential regressions and progressions
+ on the graphs tracked on v2 dashboards.
+
+ * init-database.sql: Added analysis_strategies table and task_segmentation and task_test_range
+ columns to analysis_tasks to keep the segmentation and test range selection strategies used
+ to create an analysis task.
+
+ * public/api/analysis-tasks.php:
+ (format_task): Include task_segmentation and analysis_tasks in the results.
+
+ * public/include/json-header.php:
+ (remote_user_name): Returns null when the privileged API is authenticated as a slave instead
+ of a CSRF prevention token.
+ (should_authenticate_as_slave): Added.
+ (ensure_privileged_api_data_and_token_or_slave): Added. Authenticate as a slave if slaveName
+ and slavePassword are specified. Since detect-changes.js and other slaves are not susceptible
+ to a CSRF attack, we don't need to check a CSRF token.
+
+ * public/privileged-api/create-analysis-task.php:
+ (main): Use ensure_privileged_api_data_and_token_or_slave to let detect-changes.js create new
+ analysis task. Also add or find segmentation and test range selection strategies if specified.
+
+ * public/privileged-api/create-test-group.php:
+ (main): Use ensure_privileged_api_data_and_token_or_slave.
+
+ * public/privileged-api/generate-csrf-token.php:
+
+ * public/v2/app.js:
+ (App.Pane._computeMovingAverageAndOutliers): _executeStrategy has been moved to Statistics.
+
+ * public/v2/data.js: Export Measurement, RunsData, TimeSeries. Used in detect-changes.js.
+ (Array.prototype.find): Added a polyfill to be used in node.js.
+ (RunsData.fetchRuns):
+ (RunsData.pathForFetchingRuns): Extracted from fetchRuns. Used in detect-changes.js.
+ (RunsData.createRunsDataInResponse): Extracted from App.Manifest._formatFetchedData to use it
+ in detect-changes.js.
+ (RunsData.unitFromMetricName): Ditto.
+ (RunsData.isSmallerBetter): Ditto.
+ (RunsData.prototype._timeSeriesByTimeInternal): Added secondaryTime to sort points when commit
+ times are identical.
+ (TimeSeries): When commit times are identical, order points based on build time. This is needed
+ for when we trigger two builds at two different OS versions with the same WebKit revision since
+ OS versions don't change the commit times.
+ (TimeSeries.prototype.findPointByIndex): Added.
+ (TimeSeries.prototype.rawValues): Added.
+
+ * public/v2/js/statistics.js:
+ (Statistics.TestRangeSelectionStrategies.[0]): Use the 99% two-sided probability as claimed in the
+ description of this strategy instead of the default probability. Also fixed a bug that debugging
+ code was referring to non-existent variables.
+ (Statistics.executeStrategy): Moved from App.Pane (app.js).
+
+ * public/v2/manifest.js:
+ (App.Manifest._formatFetchedData): Various code has been extracted into RunsData in data.js to be
+ used in detect-changes.js.
+
+ * tools/detect-changes.js: Added. The script fetches the manifest JSON, analyzes each graph in
+ the v2 dashboards, and creates an analysis task for the latest regression or progression detected.
+ It also schedules an A/B testing if possible and notifies another server; e.g. to send an email.
+ (main): Loads the settings JSON specified in the argument.
+ (fetchManifestAndAnalyzeData): The main loop that periodically wakes up to do the analysis.
+ (mapInOrder): Executes callback sequentially (i.e. blocking) on each item in the array.
+ (configurationsForTesting): Finds every (platform, metric) pair to analyze in the v2 dashbaords,
+ and computes various values for when statistically significant changes are detected later.
+ (analyzeConfiguration): Finds potential regressions and progression in the last X days where X
+ is the specified maximum number of days using the specified strategies. Sort the resultant ranges
+ in chronological order and create a new analysis task for the very last change we detected. We'll
+ eventually create an analysis task for all detected changes since we're repeating the analysis in
+ fetchManifestAndAnalyzeData after some time.
+ (computeRangesForTesting): Fetch measured values and compute ranges to test using the specified
+ segmentation and test range selection strategies. Once ranges are found, find overlapping analysis
+ tasks as they need to be filtered out in analyzeConfiguration to avoid creating multiple analysis
+ tasks for the same range (e.g. humans may create one before the script gets to do it).
+ (createAnalysisTaskAndNotify): Create a new analysis task for the specified range, trigger an A/B
+ testing if available, and notify another server with a HTML message as specified.
+ (findStrategyByLabel):
+ (changeTypeForRange): A change is a regression if values are getting larger in a smaller-is-better
+ test or values are getting smaller in a larger-is-better test and vice versa.
+ (summarizeRange): Create a human readable string that summarizes the change detected. e.g.
+ "Potential 3.2% regression detected between 2015-04-20 12:00 and 17:00".
+ (formatTimeRange):
+ (getJSON):
+ (postJSON):
+ (postNotification): Recursively replaces $title and $massage in the specified JSON template.
+ (instantiateNotificationTemplate):
+ (fetchJSON):
+
2015-04-20 Ryosuke Niwa <[email protected]>
Perf dashboard should have UI to set status on analysis tasks
Modified: trunk/Websites/perf.webkit.org/init-database.sql (183231 => 183232)
--- trunk/Websites/perf.webkit.org/init-database.sql 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/init-database.sql 2015-04-24 01:16:37 UTC (rev 183232)
@@ -174,11 +174,17 @@
report_failure varchar(64),
report_failure_details text);
+CREATE TABLE analysis_strategies (
+ strategy_id serial PRIMARY KEY,
+ strategy_name varchar(64) NOT NULL);
+
CREATE TYPE analysis_task_result_type as ENUM ('progression', 'regression', 'unchanged', 'inconclusive');
CREATE TABLE analysis_tasks (
task_id serial PRIMARY KEY,
task_name varchar(256) NOT NULL,
task_author varchar(256),
+ task_segmentation integer REFERENCES analysis_strategies,
+ task_test_range integer REFERENCES analysis_strategies,
task_created_at timestamp NOT NULL DEFAULT (CURRENT_TIMESTAMP AT TIME ZONE 'UTC'),
task_platform integer REFERENCES platforms NOT NULL,
task_metric integer REFERENCES test_metrics NOT NULL,
Modified: trunk/Websites/perf.webkit.org/public/api/analysis-tasks.php (183231 => 183232)
--- trunk/Websites/perf.webkit.org/public/api/analysis-tasks.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/api/analysis-tasks.php 2015-04-24 01:16:37 UTC (rev 183232)
@@ -81,6 +81,8 @@
'id' => $task_row['task_id'],
'name' => $task_row['task_name'],
'author' => $task_row['task_author'],
+ 'segmentationStrategy' => $task_row['task_segmentation'],
+ 'testRangeStragegy' => $task_row['task_test_range'],
'createdAt' => strtotime($task_row['task_created_at']) * 1000,
'platform' => $task_row['task_platform'],
'metric' => $task_row['task_metric'],
Modified: trunk/Websites/perf.webkit.org/public/include/json-header.php (183231 => 183232)
--- trunk/Websites/perf.webkit.org/public/include/json-header.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/include/json-header.php 2015-04-24 01:16:37 UTC (rev 183232)
@@ -96,10 +96,23 @@
return $data;
}
-function remote_user_name() {
- return array_get($_SERVER, 'REMOTE_USER');
+function remote_user_name($data) {
+ return should_authenticate_as_slave($data) ? NULL : array_get($_SERVER, 'REMOTE_USER');
}
+function should_authenticate_as_slave($data) {
+ return array_key_exists('slaveName', $data) && array_key_exists('slavePassword', $data);
+}
+
+function ensure_privileged_api_data_and_token_or_slave($db) {
+ $data = ""
+ if (should_authenticate_as_slave($data))
+ verify_slave($db, $data);
+ else if (!verify_token(array_get($data, 'token')))
+ exit_with_error('InvalidToken');
+ return $data;
+}
+
function compute_token() {
if (!array_key_exists('CSRFSalt', $_COOKIE) || !array_key_exists('CSRFExpiration', $_COOKIE))
return NULL;
Modified: trunk/Websites/perf.webkit.org/public/privileged-api/create-analysis-task.php (183231 => 183232)
--- trunk/Websites/perf.webkit.org/public/privileged-api/create-analysis-task.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/privileged-api/create-analysis-task.php 2015-04-24 01:16:37 UTC (rev 183232)
@@ -3,26 +3,48 @@
require_once('../include/json-header.php');
function main() {
- $data = ""
+ $db = connect();
+ $data = ""
- $author = remote_user_name();
+ $author = remote_user_name($data);
$name = array_get($data, 'name');
$start_run_id = array_get($data, 'startRun');
$end_run_id = array_get($data, 'endRun');
+ $segmentation_name = array_get($data, 'segmentationStrategy');
+ $test_range_name = array_get($data, 'testRangeStrategy');
+
if (!$name)
exit_with_error('MissingName', array('name' => $name));
$range = array('startRunId' => $start_run_id, 'endRunId' => $end_run_id);
if (!$start_run_id || !$end_run_id)
exit_with_error('MissingRange', $range);
- $db = connect();
$start_run = ensure_row_by_id($db, 'test_runs', 'run', $start_run_id, 'InvalidStartRun', $range);
$end_run = ensure_row_by_id($db, 'test_runs', 'run', $end_run_id, 'InvalidEndRun', $range);
$config = ensure_config_from_runs($db, $start_run, $end_run);
$db->begin_transaction();
+
+ $segmentation_id = NULL;
+ if ($segmentation_name) {
+ $segmentation_id = $db->select_or_insert_row('analysis_strategies', 'strategy', array('name' => $segmentation_name));
+ if (!$segmentation_id) {
+ $db->rollback_transaction();
+ exit_with_error('CannotFindOrInsertSegmentationStrategy', array('segmentationStrategy' => $segmentation_name));
+ }
+ }
+
+ $test_range_id = NULL;
+ if ($test_range_name) {
+ $test_range_id = $db->select_or_insert_row('analysis_strategies', 'strategy', array('name' => $test_range_name));
+ if (!$test_range_id) {
+ $db->rollback_transaction();
+ exit_with_error('CannotFindOrInsertTestRangeStrategy', array('testRangeStrategy' => $test_range_name));
+ }
+ }
+
$duplicate = $db->select_first_row('analysis_tasks', 'task', array('start_run' => $start_run_id, 'end_run' => $end_run_id));
if ($duplicate) {
$db->rollback_transaction();
@@ -35,7 +57,9 @@
'platform' => $config['config_platform'],
'metric' => $config['config_metric'],
'start_run' => $start_run_id,
- 'end_run' => $end_run_id));
+ 'end_run' => $end_run_id,
+ 'segmentation' => $segmentation_id,
+ 'test_range' => $test_range_id));
$db->commit_transaction();
exit_with_success(array('taskId' => $task_id));
Modified: trunk/Websites/perf.webkit.org/public/privileged-api/create-test-group.php (183231 => 183232)
--- trunk/Websites/perf.webkit.org/public/privileged-api/create-test-group.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/privileged-api/create-test-group.php 2015-04-24 01:16:37 UTC (rev 183232)
@@ -3,10 +3,10 @@
require_once('../include/json-header.php');
function main() {
- $data = ""
+ $db = connect();
+ $data = ""
+ $author = remote_user_name($data);
- $author = remote_user_name();
-
$task_id = array_get($data, 'task');
$name = array_get($data, 'name');
$root_sets = array_get($data, 'rootSets');
@@ -19,7 +19,6 @@
if ($repetition_count < 1)
exit_with_error('InvalidRepetitionCount', array('repetitionCount' => $repetition_count));
- $db = connect();
$task = $db->select_first_row('analysis_tasks', 'task', array('id' => $task_id));
if (!$task)
exit_with_error('InvalidTask', array('task' => $task_id));
Modified: trunk/Websites/perf.webkit.org/public/privileged-api/generate-csrf-token.php (183231 => 183232)
--- trunk/Websites/perf.webkit.org/public/privileged-api/generate-csrf-token.php 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/privileged-api/generate-csrf-token.php 2015-04-24 01:16:37 UTC (rev 183232)
@@ -2,7 +2,7 @@
require_once('../include/json-header.php');
-ensure_privileged_api_data();
+$data = ""
$expiritaion = time() + 3600; // Valid for one hour.
$_COOKIE['CSRFSalt'] = rand();
@@ -11,6 +11,6 @@
setcookie('CSRFSalt', $_COOKIE['CSRFSalt']);
setcookie('CSRFExpiration', $expiritaion);
-exit_with_success(array('user' => remote_user_name(), 'token' => compute_token(), 'expiration' => $expiritaion * 1000));
+exit_with_success(array('user' => remote_user_name($data), 'token' => compute_token(), 'expiration' => $expiritaion * 1000));
?>
Modified: trunk/Websites/perf.webkit.org/public/v2/app.js (183231 => 183232)
--- trunk/Websites/perf.webkit.org/public/v2/app.js 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/v2/app.js 2015-04-24 01:16:37 UTC (rev 183232)
@@ -583,19 +583,21 @@
_computeMovingAverageAndOutliers: function (chartData, movingAverageStrategy, envelopingStrategy, testRangeSelectionStrategy, anomalyDetectionStrategies)
{
var currentTimeSeriesData = chartData.current.series();
+
+ var rawValues = chartData.current.rawValues();
var movingAverageIsSetByUser = movingAverageStrategy && movingAverageStrategy.execute;
- var movingAverageValues = this._executeStrategy(
- movingAverageIsSetByUser ? movingAverageStrategy : Statistics.MovingAverageStrategies[0], currentTimeSeriesData);
+ var movingAverageValues = Statistics.executeStrategy(
+ movingAverageIsSetByUser ? movingAverageStrategy : Statistics.MovingAverageStrategies[0], rawValues);
if (!movingAverageValues)
return null;
var testRangeCandidates = [];
if (movingAverageStrategy && movingAverageStrategy.isSegmentation && testRangeSelectionStrategy && testRangeSelectionStrategy.execute)
- testRangeCandidates = this._executeStrategy(testRangeSelectionStrategy, currentTimeSeriesData, [movingAverageValues]);
+ testRangeCandidates = Statistics.executeStrategy(testRangeSelectionStrategy, rawValues, [movingAverageValues]);
var envelopeIsSetByUser = envelopingStrategy && envelopingStrategy.execute;
- var envelopeDelta = this._executeStrategy(envelopeIsSetByUser ? envelopingStrategy : Statistics.EnvelopingStrategies[0],
- currentTimeSeriesData, [movingAverageValues]);
+ var envelopeDelta = Statistics.executeStrategy(envelopeIsSetByUser ? envelopingStrategy : Statistics.EnvelopingStrategies[0],
+ rawValues, [movingAverageValues]);
for (var i = 0; i < currentTimeSeriesData.length; i++) {
var currentValue = currentTimeSeriesData[i].value;
@@ -610,7 +612,7 @@
if (anomalyDetectionStrategies.length) {
var isAnomalyArray = new Array(currentTimeSeriesData.length);
for (var strategy of anomalyDetectionStrategies) {
- var anomalyLengths = this._executeStrategy(strategy, currentTimeSeriesData, [movingAverageValues, envelopeDelta]);
+ var anomalyLengths = Statistics.executeStrategy(strategy, rawValues, [movingAverageValues, envelopeDelta]);
for (var i = 0; i < currentTimeSeriesData.length; i++)
isAnomalyArray[i] = isAnomalyArray[i] || anomalyLengths[i];
}
@@ -642,15 +644,6 @@
testRangeCandidates: testRangeCandidates,
};
},
- _executeStrategy: function (strategy, currentTimeSeriesData, additionalArguments)
- {
- var parameters = (strategy.parameterList || []).map(function (param) {
- var parsed = parseFloat(param.value);
- return Math.min(param.max || Infinity, Math.max(param.min || -Infinity, isNaN(parsed) ? 0 : parsed));
- });
- parameters.push(currentTimeSeriesData.map(function (point) { return point.value }));
- return strategy.execute.apply(window, parameters.concat(additionalArguments));
- },
_updateStrategyConfigIfNeeded: function (strategy, configName)
{
var config = null;
Modified: trunk/Websites/perf.webkit.org/public/v2/data.js (183231 => 183232)
--- trunk/Websites/perf.webkit.org/public/v2/data.js 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/v2/data.js 2015-04-24 01:16:37 UTC (rev 183232)
@@ -1,5 +1,15 @@
// We don't use DS.Model for these object types because we can't afford to process millions of them.
+if (!Array.prototype.find) {
+ Array.prototype.find = function (callback) {
+ for (var item of this) {
+ if (callback(item))
+ return item;
+ }
+ return undefined;
+ }
+}
+
var PrivilegedAPI = {
_token: null,
_expiration: null,
@@ -333,6 +343,7 @@
series.push({
measurement: measurement,
time: useCommitType ? measurement.latestCommitTime() : measurement.buildTime(),
+ secondaryTime: measurement.buildTime(),
value: measurement.mean(),
interval: measurement.confidenceInterval(),
markedOutlier: measurement.markedOutlier(),
@@ -345,28 +356,14 @@
// we don't have to fetch the entire time series to just show the last 3 days.
RunsData.fetchRuns = function (platformId, metricId, testGroupId, useCache)
{
- var url = "" ? '../data/' : '../api/runs/';
-
- url += platformId + '-' + metricId + '.json';
- if (testGroupId)
- url += '?testGroup=' + testGroupId;
-
+ var url = "" metricId, testGroupId, useCache);
return new Ember.RSVP.Promise(function (resolve, reject) {
$.getJSON(url, function (response) {
if (response.status != 'OK') {
reject(response.status);
return;
}
- delete response.status;
-
- var data = ""
- for (var config in data)
- data[config] = new RunsData(data[config]);
-
- if (response.lastModified)
- response.lastModified = new Date(response.lastModified);
-
- resolve(response);
+ resolve(RunsData.createRunsDataInResponse(response));
}).fail(function (xhr, status, error) {
if (xhr.status == 404 && useCache)
resolve(null);
@@ -376,9 +373,58 @@
});
}
+RunsData.pathForFetchingRuns = function (platformId, metricId, testGroupId, useCache)
+{
+ var path = useCache ? '/data/' : '/api/runs/';
+
+ path += platformId + '-' + metricId + '.json';
+ if (testGroupId)
+ path += '?testGroup=' + testGroupId;
+
+ return path;
+}
+
+RunsData.createRunsDataInResponse = function (response)
+{
+ delete response.status;
+
+ var data = ""
+ for (var config in data)
+ data[config] = new RunsData(data[config]);
+
+ if (response.lastModified)
+ response.lastModified = new Date(response.lastModified);
+
+ return response;
+}
+
+// FIXME: It was a mistake to put this in the client side. We should put this back in the JSON.
+RunsData.unitFromMetricName = function (metricName)
+{
+ var suffix = metricName.match('([A-z][a-z]+|FrameRate)$')[0];
+ var unit = {
+ 'FrameRate': 'fps',
+ 'Runs': '/s',
+ 'Time': 'ms',
+ 'Malloc': 'bytes',
+ 'Heap': 'bytes',
+ 'Allocations': 'bytes'
+ }[suffix];
+ return unit;
+}
+
+RunsData.isSmallerBetter = function (unit)
+{
+ return unit != 'fps' && unit != '/s';
+}
+
function TimeSeries(series)
{
- this._series = series.sort(function (a, b) { return a.time - b.time; });
+ this._series = series.sort(function (a, b) {
+ var diff = a.time - b.time;
+ return diff ? diff : a.secondaryTime - b.secondaryTime;
+ });
+
var self = this;
var min = undefined;
var max = undefined;
@@ -394,6 +440,13 @@
this._max = max;
}
+TimeSeries.prototype.findPointByIndex = function (index)
+{
+ if (!this._series || index < 0 || index >= this._series.length)
+ return null;
+ return this._series[index];
+}
+
TimeSeries.prototype.findPointByBuild = function (buildId)
{
return this._series.find(function (point) { return point.measurement.buildId() == buildId; })
@@ -462,6 +515,11 @@
TimeSeries.prototype.series = function () { return this._series; }
+TimeSeries.prototype.rawValues = function ()
+{
+ return this._series.map(function (point) { return point.value });
+}
+
TimeSeries.prototype.lastPoint = function ()
{
if (!this._series || !this._series.length)
@@ -482,3 +540,10 @@
return null;
return this._series[point.seriesIndex + 1];
}
+
+if (typeof module != 'undefined') {
+ Statistics = require('./js/statistics.js');
+ module.exports.Measurement = Measurement;
+ module.exports.RunsData = RunsData;
+ module.exports.TimeSeries = TimeSeries;
+}
Modified: trunk/Websites/perf.webkit.org/public/v2/js/statistics.js (183231 => 183232)
--- trunk/Websites/perf.webkit.org/public/v2/js/statistics.js 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/v2/js/statistics.js 2015-04-24 01:16:37 UTC (rev 183232)
@@ -487,7 +487,7 @@
for (var leftEdge = i - 2, rightEdge = i + 2; leftEdge >= 0 && rightEdge <= values.length; leftEdge--, rightEdge++) {
if (segmentedValues[leftEdge] != previousMean || segmentedValues[rightEdge - 1] != currentMean)
break;
- var result = Statistics.computeWelchsT(values, leftEdge, i - leftEdge, values, i, rightEdge - i);
+ var result = Statistics.computeWelchsT(values, leftEdge, i - leftEdge, values, i, rightEdge - i, 0.98);
if (result.significantlyDifferent) {
selectedRanges.push([leftEdge, rightEdge - 1]);
found = true;
@@ -495,7 +495,7 @@
}
}
if (!found && Statistics.debuggingTestingRangeNomination)
- console.log('Failed to find a testing range at', i, 'changing from', previousValue, 'to', currentValue);
+ console.log('Failed to find a testing range at', i, 'changing from', previousMean, 'to', currentMean);
previousMean = currentMean;
}
return selectedRanges;
@@ -565,6 +565,16 @@
},
]
+ this.executeStrategy = function (strategy, rawValues, additionalArguments)
+ {
+ var parameters = (strategy.parameterList || []).map(function (param) {
+ var parsed = parseFloat(param.value);
+ return Math.min(param.max || Infinity, Math.max(param.min || -Infinity, isNaN(parsed) ? 0 : parsed));
+ });
+ parameters.push(rawValues);
+ return strategy.execute.apply(strategy, parameters.concat(additionalArguments));
+ };
+
})();
if (typeof module != 'undefined') {
Modified: trunk/Websites/perf.webkit.org/public/v2/manifest.js (183231 => 183232)
--- trunk/Websites/perf.webkit.org/public/v2/manifest.js 2015-04-24 00:23:33 UTC (rev 183231)
+++ trunk/Websites/perf.webkit.org/public/v2/manifest.js 2015-04-24 01:16:37 UTC (rev 183232)
@@ -317,18 +317,9 @@
},
_formatFetchedData: function (metricName, configurations)
{
- var suffix = metricName.match('([A-z][a-z]+|FrameRate)$')[0];
- var unit = {
- 'FrameRate': 'fps',
- 'Runs': '/s',
- 'Time': 'ms',
- 'Malloc': 'bytes',
- 'Heap': 'bytes',
- 'Allocations': 'bytes'
- }[suffix];
+ var unit = RunsData.unitFromMetricName(metricName);
+ var smallerIsBetter = RunsData.isSmallerBetter(unit);
- var smallerIsBetter = unit != 'fps' && unit != '/s'; // Assume smaller is better for unit-less metrics.
-
var useSI = unit == 'bytes';
var unitSuffix = unit ? ' ' + unit : '';
var deltaFormatterWithoutSign = useSI ? d3.format('.2s') : d3.format('.2g');
Added: trunk/Websites/perf.webkit.org/tools/detect-changes.js (0 => 183232)
--- trunk/Websites/perf.webkit.org/tools/detect-changes.js (rev 0)
+++ trunk/Websites/perf.webkit.org/tools/detect-changes.js 2015-04-24 01:16:37 UTC (rev 183232)
@@ -0,0 +1,374 @@
+#!/usr/local/bin/node
+
+var fs = require('fs');
+var http = require('http');
+var https = require('https');
+var data = ""
+var RunsData = data.RunsData;
+var Statistics = require('../public/v2/js/statistics.js');
+
+var settings;
+function main(argv)
+{
+ if (argv.length < 3) {
+ console.error('Please specify the settings JSON path');
+ return 1;
+ }
+
+ settings = JSON.parse(fs.readFileSync(argv[2], 'utf8'));
+
+ fetchManifestAndAnalyzeData();
+}
+
+function fetchManifestAndAnalyzeData()
+{
+ getJSON(settings.perfserver, '/data/manifest.json').then(function (manifest) {
+ return mapInOrder(configurationsForTesting(manifest), analyzeConfiguration);
+ }).catch(function (reason) {
+ console.error('Failed to obtain the manifest file');
+ }).then(function () {
+ console.log('');
+ console.log('Sleeing for', settings.secondsToSleep, 'seconds');
+ setTimeout(fetchManifestAndAnalyzeData, settings.secondsToSleep * 1000);
+ });
+}
+
+function mapInOrder(array, callback, startIndex)
+{
+ if (startIndex === undefined)
+ startIndex = 0;
+ if (startIndex >= array.length)
+ return;
+
+ var next = function () { return mapInOrder(array, callback, startIndex + 1); };
+ var returnValue = callback(array[startIndex]);
+ if (typeof(returnValue) === 'object' && returnValue instanceof Promise)
+ return returnValue.then(next).catch(next);
+ return next();
+}
+
+function configurationsForTesting(manifest)
+{
+ var configurations = [];
+ for (var name in manifest.dashboards) {
+ var dashboard = manifest.dashboards[name];
+ for (var row of dashboard) {
+ for (var cell of row) {
+ if (cell instanceof Array)
+ configurations.push({platformId: parseInt(cell[0]), metricId: parseInt(cell[1])});
+ }
+ }
+ }
+
+ var platforms = manifest.all;
+ for (var config of configurations) {
+ var metric = manifest.metrics[config.metricId];
+
+ var testPath = [];
+ var id = metric.test;
+ while (id) {
+ var test = manifest.tests[id];
+ testPath.push(test.name);
+ id = test.parentId;
+ }
+
+ config.unit = RunsData.unitFromMetricName(metric.name);
+ config.smallerIsBetter = RunsData.isSmallerBetter(config.unit);
+ config.platformName = platforms[config.platformId].name;
+ config.testName = testPath.reverse().join(' > ');
+ config.fullTestName = config.testName + ':' + metric.name;
+ config.repositories = manifest.repositories;
+ if (metric.aggregator)
+ config.fullTestName += ':' + metric.aggregator;
+ }
+
+ return configurations;
+}
+
+function analyzeConfiguration(config)
+{
+ var minTime = Date.now() - settings.maxDays * 24 * 3600 * 1000;
+
+ console.log('');
+ console.log('== Analyzing the last', settings.maxDays, 'days:', config.fullTestName, 'on', config.platformName, '==');
+
+ return computeRangesForTesting(settings.perfserver, settings.strategies, config.platformId, config.metricId).then(function (ranges) {
+ var filteredRanges = ranges.filter(function (range) { return range.endTime >= minTime && !range.overlappingAnalysisTasks.length; })
+ .sort(function (a, b) { return a.endTime - b.endTime });
+
+ var summary;
+ var range;
+ for (range of filteredRanges) {
+ var summary = summarizeRange(config, range);
+ console.log('Detected:', summary);
+ }
+
+ if (!range) {
+ console.log('Nothing to analyze');
+ return;
+ }
+
+ return createAnalysisTaskAndNotify(config, range, summary);
+ });
+}
+
+function computeRangesForTesting(server, strategies, platformId, metricId)
+{
+ // FIXME: Store the segmentation strategy on the server side.
+ // FIXME: Configure each strategy.
+ var segmentationStrategy = findStrategyByLabel(Statistics.MovingAverageStrategies, strategies.segmentation.label);
+ if (!segmentationStrategy) {
+ console.error('Failed to find the segmentation strategy: ' + strategies.segmentation.label);
+ return;
+ }
+
+ var testRangeStrategy = findStrategyByLabel(Statistics.TestRangeSelectionStrategies, strategies.testRange.label);
+ if (!testRangeStrategy) {
+ console.error('Failed to find the test range selection strategy: ' + strategies.testRange.label);
+ return;
+ }
+
+ var currentPromise = getJSON(server, RunsData.pathForFetchingRuns(platformId, metricId)).then(function (response) {
+ if (response.status != 'OK')
+ throw response;
+ return RunsData.createRunsDataInResponse(response).configurations.current;
+ }, function (reason) {
+ console.error('Failed to fetch the measurements:', reason);
+ });
+
+ var analysisTasksPromise = getJSON(server, '/api/analysis-tasks?platform=' + platformId + '&metric=' + metricId).then(function (response) {
+ if (response.status != 'OK')
+ throw response;
+ return response.analysisTasks.filter(function (task) { return task.startRun && task.endRun; });
+ }, function (reason) {
+ console.error('Failed to fetch the analysis tasks:', reason);
+ });
+
+ return Promise.all([currentPromise, analysisTasksPromise]).then(function (results) {
+ var currentTimeSeries = results[0].timeSeriesByCommitTime();
+ var analysisTasks = results[1];
+ var rawValues = currentTimeSeries.rawValues();
+ var segmentedValues = Statistics.executeStrategy(segmentationStrategy, rawValues);
+
+ var ranges = Statistics.executeStrategy(testRangeStrategy, rawValues, [segmentedValues]).map(function (range) {
+ var startPoint = currentTimeSeries.findPointByIndex(range[0]);
+ var endPoint = currentTimeSeries.findPointByIndex(range[1]);
+ return {
+ startIndex: range[0],
+ endIndex: range[1],
+ overlappingAnalysisTasks: [],
+ startTime: startPoint.time,
+ endTime: endPoint.time,
+ relativeChangeInSegmentedValues: (segmentedValues[range[1]] - segmentedValues[range[0]]) / segmentedValues[range[0]],
+ startMeasurement: startPoint.measurement,
+ endMeasurement: endPoint.measurement,
+ };
+ });
+
+ for (var task of analysisTasks) {
+ var taskStartPoint = currentTimeSeries.findPointByMeasurementId(task.startRun);
+ var taskEndPoint = currentTimeSeries.findPointByMeasurementId(task.endRun);
+ for (var range of ranges) {
+ var disjoint = range.endIndex < taskStartPoint.seriesIndex
+ || taskEndPoint.seriesIndex < range.startIndex;
+ if (!disjoint)
+ range.overlappingAnalysisTasks.push(task);
+ }
+ }
+
+ return ranges;
+ });
+}
+
+function createAnalysisTaskAndNotify(config, range, summary)
+{
+ var segmentationStrategy = settings.strategies.segmentation.label;
+ var testRangeStrategy = settings.strategies.testRange.label;
+
+ var analysisTaskData = {
+ name: summary,
+ startRun: range.startMeasurement.id(),
+ endRun: range.endMeasurement.id(),
+ segmentationStrategy: segmentationStrategy,
+ testRangeStrategy: testRangeStrategy,
+
+ slaveName: settings.slave.name,
+ slavePassword: settings.slave.password,
+ };
+
+ return postJSON(settings.perfserver, '/privileged-api/create-analysis-task', analysisTaskData).then(function (response) {
+ if (response['status'] != 'OK')
+ throw response;
+
+ var analysisTaskId = response['taskId'];
+
+ var title = '[' + config.testName + '][' + config.platformName + '] ' + summary;
+ var analysisTaskURL = settings.perfserver.scheme + '://' + settings.perfserver.host + '/v2/#/analysis/task/' + analysisTaskId;
+ var changeType = changeTypeForRange(config, range);
+ // FIXME: Templatize this.
+ var message = '<b>' + settings.notification.serviceName + '</b> detected a potential ' + changeType + ':<br><br>'
+ + '<table border=1><caption>' + summary + '</caption><tbody>'
+ + '<tr><th>Test</th><td>' + config.fullTestName + '</td></tr>'
+ + '<tr><th>Platform</th><td>' + config.platformName + '</td></tr>'
+ + '<tr><th>Algorithm</th><td>' + segmentationStrategy + '<br>' + testRangeStrategy + '</td></tr>'
+ + '</table><br>'
+ + '<a href="" + analysisTaskURL + '">Open the analysis task</a>';
+
+ return getJSON(settings.perfserver, '/api/triggerables?task=' + analysisTaskId).then(function (response) {
+ var status = response['status'];
+ var triggerables = response['triggerables'] || [];
+ if (status == 'TriggerableNotFoundForTask' || triggerables.length != 1) {
+ message += ' (A/B testing was not available)';
+ return;
+ }
+ if (status != 'OK')
+ throw response;
+
+ var triggerable = response['triggerables'][0];
+ var rootSets = {};
+ for (var repositoryId of triggerable['acceptedRepositories']) {
+ var startRevision = range.startMeasurement.revisionForRepository(repositoryId);
+ var endRevision = range.endMeasurement.revisionForRepository(repositoryId);
+ if (startRevision == null || endRevision == null)
+ continue;
+ rootSets[config.repositories[repositoryId].name] = [startRevision, endRevision];
+ }
+
+ var testData = {
+ task: analysisTaskId,
+ name: 'Confirming the ' + changeType,
+ rootSets: rootSets,
+ repetitionCount: Math.max(2, Math.min(8, Math.floor((range.endIndex - range.startIndex) / 4))),
+
+ slaveName: settings.slave.name,
+ slavePassword: settings.slave.password,
+ };
+
+ return postJSON(settings.perfserver, '/privileged-api/create-test-group', testData).then(function (response) {
+ if (response['status'] != 'OK')
+ throw response;
+ message += ' (triggered an A/B testing)';
+ });
+ }).catch(function (reason) {
+ console.error(reason);
+ message += ' (failed to create a new A/B testing)';
+ }).then(function () {
+ return postNotification(settings.notification.server, settings.notification.template, title, message).then(function () {
+ console.log(' Sent a notification');
+ }, function (reason) {
+ console.error(' Failed to send a notification', reason);
+ });
+ });
+ }).catch(function (reason) {
+ console.error(' Failed to create an analysis task', reason);
+ });
+}
+
+function findStrategyByLabel(list, label)
+{
+ for (var strategy of list) {
+ if (strategy.label == label)
+ return strategy;
+ }
+ return null;
+}
+
+function changeTypeForRange(config, range)
+{
+ var endValueIsLarger = range.relativeChangeInSegmentedValues > 0;
+ return endValueIsLarger == config.smallerIsBetter ? 'regression' : 'progression';
+}
+
+function summarizeRange(config, range)
+{
+ return 'Potential ' + Math.abs(range.relativeChangeInSegmentedValues * 100).toPrecision(2) + '% '
+ + changeTypeForRange(config, range) + ' between ' + formatTimeRange(range.startTime, range.endTime);
+}
+
+function formatTimeRange(start, end)
+{
+ var formatter = function (date) { return date.toISOString().replace('T', ' ').replace(/:\d{2}\.\d+Z$/, ''); }
+ var formattedStart = formatter(start);
+ var formattedEnd = formatter(end);
+ if (start.toDateString() == end.toDateString())
+ return formattedStart + ' and ' + formattedEnd.substring(formattedEnd.indexOf(' ') + 1);
+ if (start.getFullYear() == end.getFullYear())
+ return formattedStart + ' and ' + formattedEnd.substring(5);
+ return formattedStart + ' and ' + formattedEnd;
+}
+
+function getJSON(server, path, data)
+{
+ return fetchJSON(server.scheme, {
+ 'hostname': server.host,
+ 'port': server.port,
+ 'auth': server.auth,
+ 'path': path,
+ 'method': 'GET',
+ }, 'application/json');
+}
+
+function postJSON(server, path, data)
+{
+ return fetchJSON(server.scheme, {
+ 'hostname': server.host,
+ 'port': server.port,
+ 'auth': server.auth,
+ 'path': path,
+ 'method': 'POST',
+ }, 'application/json', JSON.stringify(data));
+}
+
+function postNotification(server, template, title, message)
+{
+ var notification = instantiateNotificationTemplate(template, title, message);
+ return fetchJSON(server.scheme, {
+ 'hostname': server.host,
+ 'port': server.port,
+ 'auth': server.auth,
+ 'path': server.path,
+ 'method': server.method,
+ }, 'application/json', JSON.stringify(notification));
+}
+
+function instantiateNotificationTemplate(template, title, message)
+{
+ var instance = {};
+ for (var name in template) {
+ var value = template[name];
+ if (typeof(value) === 'string')
+ instance[name] = value.replace(/\$title/g, title).replace(/\$message/g, message);
+ else if (typeof(template[name]) === 'object')
+ instance[name] = instantiateNotificationTemplate(value, title, message);
+ else
+ instance[name] = value;
+ }
+ return instance;
+}
+
+function fetchJSON(schemeName, options, contentType, content) {
+ var requester = schemeName == 'https' ? https : http;
+ return new Promise(function (resolve, reject) {
+ var request = requester.request(options, function (response) {
+ var responseText = '';
+ response.setEncoding('utf8');
+ response.on('data', function (chunk) { responseText += chunk; });
+ response.on('end', function () {
+ try {
+ var json = JSON.parse(responseText);
+ } catch (error) {
+ reject({error: error, responseText: responseText});
+ }
+ resolve(json);
+ });
+ });
+ request.on('error', function (error) { reject(error); });
+ if (contentType)
+ request.setHeader('Content-Type', contentType);
+ if (content)
+ request.write(content);
+ request.end();
+ });
+}
+
+main(process.argv);
