Repository: yetus Updated Branches: refs/heads/master 8829c2b41 -> f94eff37c
YETUS-594. Fix up the jenkins precommit script Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Vihang Karajgaonkar <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/yetus/repo Commit: http://git-wip-us.apache.org/repos/asf/yetus/commit/f94eff37 Tree: http://git-wip-us.apache.org/repos/asf/yetus/tree/f94eff37 Diff: http://git-wip-us.apache.org/repos/asf/yetus/diff/f94eff37 Branch: refs/heads/master Commit: f94eff37c3d05801c33cda0f831269f91bc3a82d Parents: 8829c2b Author: Allen Wittenauer <[email protected]> Authored: Mon Dec 11 09:38:01 2017 -0800 Committer: Allen Wittenauer <[email protected]> Committed: Tue Jan 16 17:43:54 2018 -0800 ---------------------------------------------------------------------- .../in-progress/precommit-admin.md | 84 +++++ build.sh | 3 +- precommit/jenkins/jenkins-admin.py | 303 ++++++++++++------- 3 files changed, 280 insertions(+), 110 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/yetus/blob/f94eff37/asf-site-src/source/documentation/in-progress/precommit-admin.md ---------------------------------------------------------------------- diff --git a/asf-site-src/source/documentation/in-progress/precommit-admin.md b/asf-site-src/source/documentation/in-progress/precommit-admin.md new file mode 100644 index 0000000..49f13eb --- /dev/null +++ b/asf-site-src/source/documentation/in-progress/precommit-admin.md @@ -0,0 +1,84 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +Overview +======== + +The jenkins-admin script is an automated way to submit JIRA issues to the Apache Yetus precommit testing framework. It works by: + +* Grab the XML output of a JIRA filter that contains all the issues that should be tested by the system. In general, the filter should comprise issues from relevant JIRA projects that are currently in Patch Available state. +* Process the XML into a list of `<project-issue>, <attachment id>` pairs, where attachment id is the id for the _newest_ attachment on the issue. Pulling the newest attachment implies that if multiple attachments are uploaded to an issue, the last one is the one that will be processed and other attachments will be _ignored_. +* Grab the single build artifact that this job keeps, `patch_tested.txt`. This file contains the list of `<project-issue>, <attachment id>` pairs that have already been submitted for testing. +* For each pair from the processed XML, see if the pair is in `patch_tested.txt`. If not, start a new project build and append the pair to the file. +* When this admin job completes, archive the latest `patch_tested.txt` file. + +All communication to Jenkins is currently done with the Remote Access API using tokens where necessary (see below). JIRA communication is currently unauthenticated. + +The JIRA filter is a required value provided via the `--jira-filter` parameter. It should be the full URL as provided by the JIRA UI. + +In order to prevent accidents, the `--live` parameter is required when jenkins-admin is used for production purposes. Without this flag, jobs are not launched but the `patch_tested.txt` file is created/updated. + +By default, jenkins-admin expects that the `JOB_NAME` environment variable will be set by Jenkins. If it is not or it needs to be overridden, that can be done with the `--jenkins-jobname` parameter. + +Additionally, the URL of the Jenkins server is expected to be in the `JENKINS_URL` environment variable, also usually set by Jenkins. This value may be overridden with the `--jenkins-url` option. + +The very first run of the job should be done with the `--initialize` parameter to create the first `patch_tested.txt` file. Otherwise, the job will fail because a previous version of it cannot be downloaded from previous runs. + +Project-Specifc Builds +======================= + +New builds are started via buildWithParameters call. Three parameters are added to the URL: + + * token = Jenkins security token (see below) + * ISSUE\_NUM = JIRA issue number + * ATTACHMENT\_ID = JIRA attachment id + + By default, the Jenkins job name is expected to be `PreCommit-{project}`, where the project name matches the JIRA project name. Using the JIRA issue YETUS-1 with an attachment number of 2345 would result in the following URL: + + {JENKINS\_URL}/job/PreCommit-YETUS/buildWithParameters?token=YETUS-token&ISSUE_NUM=1&ATTACHMENT_ID=2345 + + The `{JENKINS_URL}` can be overridden on a per project basis using the `--jenkins-url-override` option. This parameter allows for one job on one Jenkins server to direct different projects to different Jenkins servers. For example: + + jenkins-admin --jenkins-url-override=PROJ1=https://example.com/1 --jenkins-url-override=PROJ2=https://example.com/1 + +would send all PROJ1 Jenkins jobs to the first URL and all PROJ2 jobs to the second URL. The `--jenkins-url-override` option may be listed as many times as necessary. + +The job name can be overridden via the `--jenkins-project-template` option. For example, using `--jenkins-project-template='{project}-Build'`would change the above URL to be: + + .../job/PreCommit-YETUS-Build/buildwithParameters?... + +Jenkins Job Tokens +================== + +Currently, jenkins-admin supports the usage of Jenkins tokens for authentication via the `--jenkins-token` option. This option provides two ways to do tokens + + * Flat tokens + * Template tokens + +Flat tokens are a simple string. For example, `--jenkins-tokens=yetus` would require the `yetus` string to be listed as the token in the Jenkins job configuration that is being requested. + +On the other hand, template tokens perform some simple string substitution before being used. This exchange includes: + + * {project} for the JIRA project name + * {issue} for the JIRA issue number + * {attachment} for the JIRA issue attachment id + +For example, if JIRA issue YETUS-1 has an attachment of 2345, then `{project}` becomes `YETUS`, `{issue}` becomes `1`, and `{attachment}` becomes 2345. + +By default, the token is set to `{project}-token`. The token then becomes `YETUS-token` using the above values. http://git-wip-us.apache.org/repos/asf/yetus/blob/f94eff37/build.sh ---------------------------------------------------------------------- diff --git a/build.sh b/build.sh index 6307ce9..57920ed 100755 --- a/build.sh +++ b/build.sh @@ -216,7 +216,8 @@ for utility in shelldocs/shelldocs.py \ precommit/docker-cleanup.sh \ precommit/qbt.sh \ precommit/smart-apply-patch.sh \ - precommit/test-patch.sh + precommit/test-patch.sh \ + precommit/jenkins/jenkins-admin.py do wrapper=${utility##*/} wrapper=${wrapper%.*} http://git-wip-us.apache.org/repos/asf/yetus/blob/f94eff37/precommit/jenkins/jenkins-admin.py ---------------------------------------------------------------------- diff --git a/precommit/jenkins/jenkins-admin.py b/precommit/jenkins/jenkins-admin.py index 0fe34b0..62f6e7a 100755 --- a/precommit/jenkins/jenkins-admin.py +++ b/precommit/jenkins/jenkins-admin.py @@ -18,126 +18,211 @@ # under the License. # -# Script uses a JIRA search filter to submit test Jenkins -# jobs for patch avilable issues. For more information see: -# http://wiki.apache.org/general/PreCommitBuilds -# https://builds.apache.org/job/PreCommit-Admin/ - from optparse import OptionParser from tempfile import NamedTemporaryFile from xml.etree import ElementTree -import os, re, sys, urllib2 - -# max number of entries to keep in the patch_tested.txt file -MAX_HISTORY = 5000 +import base64 +import httplib +import os +import re +import sys +import urllib2 -def httpGet(resource, ignoreError=False): - if ignoreError: +def httpGet(resource, ignoreError=False, username=None, password=None): + request = urllib2.Request(resource) + if username and password: + base64string = base64.b64encode('%s:%s' % (username, password)) + request.add_header("Authorization", "Basic %s" % base64string) try: - return urllib2.urlopen(resource).read() - except urllib2.URLError, e: - print "ERROR retrieving resource %s: %s" % (resource, e) - return "" - return urllib2.urlopen(resource).read() + response = urllib2.urlopen(request) + except urllib2.HTTPError, http_err: + code = http_err.code + print '%s returns HTTP error %d: %s' \ + % (resource, code, http_err.reason) + if ignoreError: + return '' + else: + print 'Aborting.' + sys.exit(1) + except urllib2.URLError, url_err: + print 'Error contacting %s: %s' % (resource, url_err.reason) + if ignoreError: + return '' + else: + raise url_err + except httplib.BadStatusLine, err: + if ignoreError: + return '' + else: + raise err + return response.read() # returns a map of (project, issue) => attachment id + def parseJiraData(fileName): - tree = ElementTree.parse(fileName) - root = tree.getroot() - jiraPattern = re.compile('([A-Z]+)\-([0-9]+)') - result = {} - for item in root.findall("./channel/item"): - key = item.find('key') - if key == None: continue - issue = key.text - matcher = jiraPattern.match(issue) - if not matcher: continue - issue = matcher.group(1), matcher.group(2) - attachmentIds = [] - for attachment in item.findall('./attachments/attachment'): - attachmentId = attachment.get('id') - try: - attachmentIds.append(int(attachmentId)) - except ValueError: - pass - if len(attachmentIds) > 0: - attachmentIds.sort() - result[issue] = attachmentIds[-1] - return result + tree = ElementTree.parse(fileName) + root = tree.getroot() + jiraPattern = re.compile('([A-Z]+)\-([0-9]+)') + result = {} + for item in root.findall('./channel/item'): + jirakey = item.find('key') + if jirakey is None: + continue + jiraissue = jirakey.text + matcher = jiraPattern.match(jiraissue) + if not matcher: + continue + jiraissue = (matcher.group(1), matcher.group(2)) + attachmentIds = [] + for jiraattachment in item.findall('./attachments/attachment'): + attachmentId = jiraattachment.get('id') + try: + attachmentIds.append(int(attachmentId)) + except ValueError: + pass + if attachmentIds: + attachmentIds.sort() + result[jiraissue] = attachmentIds[-1] + return result + if __name__ == '__main__': - parser = OptionParser() - parser.add_option("--jenkins-url", dest="jenkinsUrl", - help="Jenkins base URL", metavar="URL") - parser.add_option("--jenkins-token", dest="jenkinsToken", - help="Jenkins Token", metavar="TOKEN") - parser.add_option("--jira-filter", dest="jiraFilter", - help="JIRA filter URL", metavar="URL") - parser.add_option("--jenkins-url-override", dest="jenkinsUrlOverrides", action="append", - help="Project specific Jenkins base URL", metavar="PROJECT=URL") - parser.add_option("--live", dest="live", action="store_true", - help="Submit Job to jenkins") - (options, args) = parser.parse_args() - if not options.jiraFilter: - parser.error("JIRA Filter is a required argument") - if not options.jenkinsUrl: - parser.error("Jenkins URL is a required argument") - if options.live and not options.jenkinsToken: - parser.error("Jenkins Token is required when in live mode") - jenkinsUrlOverrides = {} - if options.jenkinsUrlOverrides: - for override in options.jenkinsUrlOverrides: - if "=" not in override: - parser.error("Invalid Jenkins Url Override: " + override) - (project, url) = override.split("=", 1) - jenkinsUrlOverrides[project.upper()] = url - tempFile = NamedTemporaryFile(delete=False) - try: - jobLogHistory = httpGet(options.jenkinsUrl + \ - "/job/PreCommit-Admin/lastSuccessfulBuild/artifact/patch_tested.txt", True) - # if we don't have a successful build available try the last build - if not jobLogHistory: - jobLogHistory = httpGet(options.jenkinsUrl + \ - "/job/PreCommit-Admin/lastCompletedBuild/artifact/patch_tested.txt") - jobLogHistory = jobLogHistory.strip().split("\n") - if "TESTED ISSUES" not in jobLogHistory[0]: - print "Downloaded patch_tested.txt control file may be corrupted. Failing." - sys.exit(1) - jobLog = open('patch_tested.txt', 'w+') - if len(jobLogHistory) > MAX_HISTORY: - jobLogHistory = [ jobLogHistory[0] ] + jobLogHistory[len(jobLogHistory) - MAX_HISTORY:] - for jobHistoryRecord in jobLogHistory: - jobLog.write(jobHistoryRecord + "\n") - jobLog.flush() - rssData = httpGet(options.jiraFilter) - tempFile.write(rssData) - tempFile.flush() - for key, attachment in parseJiraData(tempFile.name).items(): - (project, issue) = key - if jenkinsUrlOverrides.has_key(project): - url = jenkinsUrlOverrides[project] - else: - url = options.jenkinsUrl - jenkinsUrlTemplate = url + "/job/PreCommit-{project}-Build/buildWithParameters" + \ - "?token={token}&ISSUE_NUM={issue}&ATTACHMENT_ID={attachment}" - urlArgs = {'token': options.jenkinsToken, 'project': project, 'issue': issue, 'attachment': attachment } - jenkinsUrl = jenkinsUrlTemplate.format(**urlArgs) - # submit job - jobName = "%s-%s,%s" % (project, issue, attachment) - if jobName not in jobLogHistory: - print jobName + " has not been processed, submitting" - jobLog.write(jobName + "\n") + parser = OptionParser() + if os.getenv('JENKINS_URL'): + parser.set_defaults(jenkinsUrl=os.getenv('JENKINS_URL')) + if os.getenv('JOB_NAME'): + parser.set_defaults(jenkinsJobName=os.getenv('JOB_NAME')) + else: + parser.set_defaults(jenkinsJobName='PreCommit-Admin') + + parser.set_defaults(jenkinsJobTemplate='PreCommit-{project}') + parser.add_option('--initialize', action='store_true', + dest='jenkinsInit', + help='Start a new patch_tested.txt file') + parser.add_option('--jenkins-jobname', type='string', + dest='jenkinsJobName', + help='PreCommit-Admin JobName', metavar='JOB_NAME') + parser.add_option('--jenkins-project-template', type='string', + dest='jenkinsJobTemplate', + help='Template for project jobs', + metavar='TEMPLATE') + parser.add_option('--jenkins-token', type='string', + dest='jenkinsToken', help='Jenkins Token', + metavar='TOKEN') + parser.add_option('--jenkins-url', type='string', dest='jenkinsUrl' + , help='Jenkins base URL', metavar='URL') + parser.add_option( + '--jenkins-url-override', + type='string', + dest='jenkinsUrlOverrides', + action='append', + help='Project specific Jenkins base URL', + metavar='PROJECT=URL', + ) + parser.add_option('--jira-filter', type='string', dest='jiraFilter', + help='JIRA filter URL', metavar='URL') + parser.add_option('--jira-user', type='string', dest='jiraUser', + help='JIRA username') + parser.add_option('--jira-password', type='string', dest='jiraPassword', + help='JIRA password') + parser.add_option('--live', dest='live', action='store_true', + help='Submit Job to jenkins') + parser.add_option('--max-history', dest='history', type='int', + help='Maximum history to store', default=5000) + (options, args) = parser.parse_args() + tokenFrag = '' + if options.jenkinsToken: + tokenFrag = 'token=%s' % options.jenkinsToken + else: + tokenFrag = 'token={project}-token' + if not options.jiraFilter: + parser.error('ERROR: --jira-filter is a required argument.') + if not options.jenkinsUrl: + parser.error('ERROR: --jenkins-url or the JENKINS_URL environment variable is required.' + ) + if options.history < 0: + parser.error('ERROR: --max-history must be 0 or a positive integer.' + ) + jenkinsUrlOverrides = {} + if options.jenkinsUrlOverrides: + for override in options.jenkinsUrlOverrides: + if '=' not in override: + parser.error('Invalid Jenkins Url Override: ' + + override) + (project, url) = override.split('=', 1) + jenkinsUrlOverrides[project.upper()] = url + tempFile = NamedTemporaryFile(delete=False) + try: + jobLogHistory = None + if not options.jenkinsInit: + jobLogHistory = httpGet(options.jenkinsUrl + + '/job/%s/lastSuccessfulBuild/artifact/patch_tested.txt' + % options.jenkinsJobName, True) + + # if we don't have a successful build available try the last build + + if not jobLogHistory: + jobLogHistory = httpGet(options.jenkinsUrl + + '/job/%s/lastCompletedBuild/artifact/patch_tested.txt' + % options.jenkinsJobName) + jobLogHistory = jobLogHistory.strip().split('\n') + if 'TESTED ISSUES' not in jobLogHistory[0]: + print 'Downloaded patch_tested.txt control file may be corrupted. Failing.' + sys.exit(1) + + # we are either going to write a new one or rewrite the old one + + jobLog = open('patch_tested.txt', 'w+') + + if jobLogHistory: + if len(jobLogHistory) > options.history: + jobLogHistory = [jobLogHistory[0]] \ + + jobLogHistory[len(jobLogHistory) + - options.history:] + for jobHistoryRecord in jobLogHistory: + jobLog.write(jobHistoryRecord + '\n') + else: + jobLog.write('TESTED ISSUES\n') jobLog.flush() + rssData = httpGet(options.jiraFilter,False,options.jiraUser,options.jiraPassword) + tempFile.write(rssData) + tempFile.flush() + for (key, attachment) in parseJiraData(tempFile.name).items(): + (project, issue) = key + if jenkinsUrlOverrides.has_key(project): + url = jenkinsUrlOverrides[project] + else: + url = options.jenkinsUrl + + jenkinsUrlTemplate = url + '/job/' \ + + options.jenkinsJobTemplate \ + + '/buildWithParameters?' + tokenFrag \ + + '&ISSUE_NUM={issue}&ATTACHMENT_ID={attachment}' + + urlArgs = { + 'project': project, + 'issue': issue, + 'attachment': attachment, + } + jenkinsUrl = jenkinsUrlTemplate.format(**urlArgs) + + # submit job + + jobName = '%s-%s,%s' % (project, issue, attachment) + if not jobLogHistory or jobName not in jobLogHistory: + print jobName + ' has not been processed, submitting' + jobLog.write(jobName + '\n') + jobLog.flush() + if options.live: + httpGet(jenkinsUrl, True) + else: + print 'GET ' + jenkinsUrl + else: + print jobName + ' has been processed, ignoring' + jobLog.close() + finally: if options.live: - httpGet(jenkinsUrl, True) + os.remove(tempFile.name) else: - print "GET " + jenkinsUrl - else: - print jobName + " has been processed, ignoring" - jobLog.close() - finally: - if options.live: - os.remove(tempFile.name) - else: - print "JIRA Data is located: " + tempFile.name + print 'JIRA Data is located: ' + tempFile.name
