Hashar has uploaded a new change for review. https://gerrit.wikimedia.org/r/189938
Change subject: browsertests: move user/pass to Credentials plugin ...................................................................... browsertests: move user/pass to Credentials plugin Our browsertests login to beta cluster and production wikis. The credentials are hold in Jenkins global configuration as environement variables which are injected in every job. Since some job might dump all environement variable, the passwords can end up being leaked publicly. I have filled user/pass in the Jenkins credentials store: https://integration.wikimedia.org/ci/credential-store/domain/browsertests/ - [email protected] - [email protected] - [email protected] - [email protected] - [email protected] - [email protected] - [email protected] Each are given an id, '@' has been replaced by '-at-'. * Make the browser tests jobs to use the Jenkins Credentials Binding plugin to fetch the user/pass. This is done using the JJB wrapper 'credentials-binding' which expose the user and pass joined with ':' as the MEDIAWIKI_CREDENTIALS variable. * Use `cut` to populate the MEDIAWIKI_USER and MEDIAWIKI_PASSWORD variables. * Hardcode MEDIAWIKI_PASSWORD_VARIABLE="MEDIAWIKI_PASSWORD" * Phase out JJB variables mediawiki_user and mediawiki_password in favor of the new mediawiki_credentials_id. The ids correspond to entries in the Credentials store. This way we restrict the list of jobs that can potentially leak the information. set +x should be enough to hide them. NOTE: the JJB credentials-binding wrapper does not support exposing the user and password in two different variables, hence the `cut` magic. The plugin does support exposing two variables but we will need to update JJB for it. Bug: T89226 Change-Id: I68fe8240f339661efecb62c32ae2da8a2d84d2c9 --- M jjb/browsertests.yaml M jjb/job-templates-browsertests.yaml M jjb/macro-browsertests.yaml 3 files changed, 74 insertions(+), 73 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/integration/config refs/changes/38/189938/1 diff --git a/jjb/browsertests.yaml b/jjb/browsertests.yaml index ecd1862..c5174d5 100644 --- a/jjb/browsertests.yaml +++ b/jjb/browsertests.yaml @@ -41,11 +41,10 @@ defaults: browsertests folder: tests headless: 'false' - mediawiki_user: Selenium_user platform: linux recipients: *emails-qa repository: mediawiki/extensions/CentralNotice - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org jobs: - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': @@ -84,18 +83,17 @@ browser: firefox folder: tests headless: 'false' - mediawiki_user: Selenium_user platform: linux recipients: *emails-CirrusSearch repository: mediawiki/extensions/CirrusSearch jobs: - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org # Core @@ -105,9 +103,8 @@ browser: firefox folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org platform: linux recipients: *emails-qa repository: mediawiki/core @@ -121,7 +118,6 @@ defaults: browsertests folder: tests headless: 'false' - mediawiki_user: Selenium_user platform: linux recipients: *emails-qa repository: mediawiki/extensions/Echo @@ -130,22 +126,22 @@ - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: chrome mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: firefox mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: chrome + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: firefox + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG # Flow - project: @@ -153,7 +149,6 @@ defaults: browsertests folder: tests headless: 'false' - mediawiki_user: Selenium_user platform: linux recipients: *emails-qa repository: mediawiki/extensions/Flow @@ -161,48 +156,45 @@ jobs: - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: chrome - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: firefox - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: internet_explorer - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org platform: 'windows_8' - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: chrome - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: firefox - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: internet_explorer platform: 'windows_8' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-monobook-sauce': browser: chrome - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user_monobook-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user_monobook - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-monobook-sauce': browser: firefox - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user_monobook-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user_monobook - @@ -212,9 +204,8 @@ defaults: browsertests folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-qa repository: mediawiki/extensions/Math @@ -233,9 +224,8 @@ browser: firefox folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-growth repository: mediawiki/extensions/GettingStarted @@ -249,9 +239,8 @@ defaults: browsertests folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.m.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-qa repository: mediawiki/extensions/MobileFrontend @@ -270,7 +259,7 @@ - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: firefox - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.m.wikipedia.org # MultimediaViewer @@ -280,9 +269,8 @@ browser: firefox folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-multimedia repository: mediawiki/extensions/MultimediaViewer @@ -323,7 +311,7 @@ - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: firefox - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: mediawiki.org # PageTriage @@ -332,9 +320,8 @@ defaults: browsertests folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-qa repository: mediawiki/extensions/PageTriage @@ -353,9 +340,8 @@ browser: firefox folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-qa repository: mediawiki/extensions/PdfHandler @@ -376,37 +362,33 @@ jobs: - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': - mediawiki_password_variable: MEDIAWIKI_PASSWORD_ULS_WMFLABS_ORG + mediawiki_credentials_id: Uls-at-beta.wmflabs.org mediawiki_url: commons.wikimedia.beta.wmflabs.org - mediawiki_user: Uls - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: language-browsertests.wmflabs.org - mediawiki_user: Selenium_user # Restrict emails notifications since language-browsertests is unstable/broken recipients: *emails-qa-alerts - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_SANDBOX_TRANSLATEWIKI_NET + mediawiki_credentials_id: Selenium-at-sandbox.translatewiki.net mediawiki_url: sandbox.translatewiki.net - mediawiki_user: Selenium # UploadWizard - project: name: UploadWizard-api defaults: browsertests - mediawiki_user: Selenium_user repository: mediawiki/extensions/UploadWizard jobs: - 'UploadWizard-api-{mediawiki_url}': - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: commons.wikimedia.beta.wmflabs.org pollscm: '* * * * *' - 'UploadWizard-api-{mediawiki_url}': - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: commons.wikimedia.org pollscm: '0 5 31 2 *' @@ -415,9 +397,8 @@ defaults: browsertests folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: commons.wikimedia.beta.wmflabs.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-UploadWizard repository: mediawiki/extensions/UploadWizard @@ -435,9 +416,8 @@ defaults: browsertests folder: modules/ve-mw/tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-VisualEditor repository: mediawiki/extensions/VisualEditor @@ -471,18 +451,18 @@ - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: chrome - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: firefox - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: internet_explorer platform: 'windows_8' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-{version}-sauce': @@ -493,7 +473,7 @@ - 'browsertests-{name}-{mediawiki_url}-{platform}-{browser}-sauce': browser: safari platform: 'os_x_10.9' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: test2.wikipedia.org - 'browsertests-{name}-production-{platform}-{browser}-sauce': @@ -501,8 +481,7 @@ - 'browsertests-{name}-language-screenshot-{platform}-{browser}': browser: firefox - mediawiki_password_variable: MEDIAWIKI_PASSWORD_LANGUAGESCREENSHOTBOT_WMFLABS_ORG - mediawiki_user: LanguageScreenshotBot + mediawiki_credentials_id: LanguageScreenshotBot-at-beta.wmflabs.org platform: 'os_x_10.10' # Wikidata @@ -512,9 +491,8 @@ browser: firefox folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: WikidataTester-at-beta.wmflabs.org mediawiki_url: wikidata.beta.wmflabs.org - mediawiki_user: WikidataTester platform: linux recipients: *emails-Wikidata-qa repository: WikidataBrowserTests.git @@ -541,9 +519,8 @@ browser: firefox folder: tests headless: 'false' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG + mediawiki_credentials_id: Selenium_user-at-beta.wmflabs.org mediawiki_url: en.wikipedia.beta.wmflabs.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-qa repository: mediawiki/extensions/WikiLove @@ -558,9 +535,8 @@ browser: phantomjs folder: tests headless: 'true' - mediawiki_password_variable: MEDIAWIKI_PASSWORD_SELENIUM_USER_WIKIPEDIA_ORG + mediawiki_credentials_id: Selenium_user-at-wikipedia.org mediawiki_url: en.m.wikipedia.org - mediawiki_user: Selenium_user platform: linux recipients: *emails-qa repository: mediawiki/extensions/ZeroBanner diff --git a/jjb/job-templates-browsertests.yaml b/jjb/job-templates-browsertests.yaml index 920c4a8..efa2ac1 100644 --- a/jjb/job-templates-browsertests.yaml +++ b/jjb/job-templates-browsertests.yaml @@ -109,9 +109,8 @@ cucumber_tags: '{cucumber_tags}' headless: '{headless}' folder: '{folder}' - mediawiki_password_variable: '{mediawiki_password_variable}' + mediawiki_credentials_id: '{mediawiki_credentials_id}' mediawiki_url: '{mediawiki_url}' - mediawiki_user: '{mediawiki_user}' platform: '{platform}' version: '{version}' @@ -139,6 +138,15 @@ wrappers: - ansicolor - timestamps + # Wiki usernames and passwords are hold in Jenkins credentials store + # https://integration.wikimedia.org/ci/credential-store/domain/browsertests/ + - credentials-binding: + - username-password: + credential-id: '{mediawiki_credentials_id}' + # FIXME JJB does not support splitted user/pass variables although the plugin does. + # http://ci.openstack.org/jenkins-job-builder/wrappers.html#wrappers.credentials-binding + # Pass both in a single variable, separated by ':' + variable: MEDIAWIKI_CREDENTIALS # UploadWizard @@ -146,7 +154,7 @@ name: 'UploadWizard-api-{mediawiki_url}' defaults: browsertests node: contintLabsSlave && UbuntuPrecise - mediawiki_password_variable: '{mediawiki_password_variable}' + mediawiki_credentials_id: '{mediawiki_credentials_id}' mediawiki_url: '{mediawiki_url}' pollscm: '{pollscm}' @@ -156,9 +164,8 @@ builders: - UploadWizard-api: - mediawiki_password_variable: '{mediawiki_password_variable}' + mediawiki_credentials_id: '{mediawiki_credentials_id}' mediawiki_url: '{mediawiki_url}' - mediawiki_user: '{mediawiki_user}' publishers: - email: @@ -171,7 +178,15 @@ - shell: | # set up environment variables set -e - export MEDIAWIKI_PASSWORD_VARIABLE={mediawiki_password_variable} + + set +x + if [ -z $MEDIAWIKI_CREDENTIALS ]; then + echo "\$MEDIAWIKI_CREDENTIALS is empty. Check job configuration." + exit 1 + fi + export MEDIAWIKI_USER=`echo $MEDIAWIKI_CREDENTIALS | cut -f1 -d:` + MEDIAWIKI_PASSWORD=`echo $MEDIAWIKI_CREDENTIALS | cut -f2- -d:` + export MEDIAWIKI_PASSWORD_VARIABLE='MEDIAWIKI_PASSWORD' # install python virtualenv --distribute DEV @@ -181,11 +196,13 @@ # run tests DEV/bin/python tests/api/upload-wizard_tests.py \ - --username "{mediawiki_user}" \ + # FIXME + --username "$MEDIAWIKI_USER" \ --api_url "http://{mediawiki_url}/w/api.php"; DEV/bin/python tests/api/upload-wizard_tests.py \ --gen_new_image \ - --username "{mediawiki_user}" \ + # FIXME + --username "$MEDIAWIKI_USER" \ --api_url "http://{mediawiki_url}/w/api.php"; # VisualEditor diff --git a/jjb/macro-browsertests.yaml b/jjb/macro-browsertests.yaml index b922d6e..95eccf3 100644 --- a/jjb/macro-browsertests.yaml +++ b/jjb/macro-browsertests.yaml @@ -11,15 +11,23 @@ export CUCUMBER_TAGS={cucumber_tags} export HEADLESS={headless} export MEDIAWIKI_API_URL=http://{mediawiki_url}/w/api.php - export MEDIAWIKI_PASSWORD_VARIABLE={mediawiki_password_variable} export MEDIAWIKI_URL=http://{mediawiki_url}/wiki/ - export MEDIAWIKI_USER={mediawiki_user} export PLATFORM='{platform}' # Replace PLATFORM underscores (Jenkins) to spaces (SauceLabs) export PLATFORM=${{PLATFORM//_/ }} export SCREENSHOT_FAILURES=true export SCREENSHOT_FAILURES_PATH="$WORKSPACE/log" + set +x + if [ -z $MEDIAWIKI_CREDENTIALS ]; then + echo "\$MEDIAWIKI_CREDENTIALS is empty. Check job configuration." + exit 1 + fi + export MEDIAWIKI_USER=`echo $MEDIAWIKI_CREDENTIALS | cut -f1 -d:` + MEDIAWIKI_PASSWORD=`echo $MEDIAWIKI_CREDENTIALS | cut -f2- -d:` + export MEDIAWIKI_PASSWORD_VARIABLE='MEDIAWIKI_PASSWORD' + set -x + # We only care about one version of our browser and do not need a job # per version. Thus the versions to use are hardcoded there. # -- To view, visit https://gerrit.wikimedia.org/r/189938 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I68fe8240f339661efecb62c32ae2da8a2d84d2c9 Gerrit-PatchSet: 1 Gerrit-Project: integration/config Gerrit-Branch: master Gerrit-Owner: Hashar <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
