Repository: incubator-gearpump Updated Branches: refs/heads/master 13ae50fb5 -> 328063f5a
gearpump-34 update developer document. Author: manuzhang <[email protected]> Author: Weihua Jiang <[email protected]> Closes #1 from whjiang/gearpump-34. Project: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/commit/328063f5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/tree/328063f5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/diff/328063f5 Branch: refs/heads/master Commit: 328063f5a172b61723263fe0ed4f2d07b63ee991 Parents: 13ae50f Author: manuzhang <[email protected]> Authored: Tue May 3 11:47:30 2016 +0800 Committer: manuzhang <[email protected]> Committed: Tue May 3 11:47:30 2016 +0800 ---------------------------------------------------------------------- CONTRIBUTING.md | 335 +++++++++++++++++++++++++++----- dev-tools/merge_gearpump_pr.py | 378 ++++++++++++++++++++++++++++++++++++ 2 files changed, 668 insertions(+), 45 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/328063f5/CONTRIBUTING.md ---------------------------------------------------------------------- diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a39da53..adaa313 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,91 +1,336 @@ -This documents are for committers which have direct write permission to intel-hadoop/gearpump +# Developer documentation -Commit Guideline -======================== -1. For all commit(except doc), must create an issue id. -2. For all commit log message, must contain the issue id, Like this: fix #issueId, comments. -3. For all PR, the title must contains issue Id. -4. We use rebase and squash instead of merge to ensure the log message is clean. Check section "Pull Request merge process for Gearpump" -5. Every commit (except doc) must have 1 guy to review before commit. +This document summarizes the information relevant to Gearpump committers and contributors. It includes information about +the development processes and policies as well as the tools we use to facilitate those. -Pull Request merge process for Gearpump -======================== -1. Fork in github to create a /gearpump repo. After fork, you will have a new repo at http://github.com/<git-user_id>/gearpump. -2. Add gearpump/gearpump as an external repo 'upstream' by following the [guide](https://help.github.com/articles/configuring-a-remote-for-a-fork/). +--- - ```bash - git remote add upstream https://github.com/gearpump/gearpump.git - ``` +Table of Contents +* <a href="#welcome">Welcome!</a> +* <a href="#workflow">Contribution workflow</a> + * <a href="#report-bug">Report a bug</a> + * <a href="#request-feature">Request a new feature</a> + * <a href="#contribute-code">Contribute code/document by creating a Pull Request</a> + * <a href="code-review">Code Review</a> +* <a href="#build-and-test">Build the code and run the tests</a> + * <a href="#local-copy">Make a local copy of Gearpump</a> + * <a href="#build">How to build</a> + * <a href="#test">How to test</a> + * <a href="#build-doc">How to build document</a> + * <a href="#ide-setup">IDE setup</a> + * <a href="#code-style">Code style</a> + * <a href="#write-unittest">How to write unit test</a> + * <a href="#write-integrationtest">How to write integration test</a> + * <a href="#write-doc">How to write document</a> +* <a href="#committer-work">Committer section</a> + * <a href="#approve-pull-request">Approve a pull request</a> + * <a href="#merge-pull-request">Merge a pull request or patch</a> + * <a href="#release">How to make a release</a> -3. In local master branch, periodically sync the forked master with the main master with - - ``` - git pull --rebase upstream master - git push origin master - ``` -No work should ever be done in the forked master. Another way to do this is to +--- + +<a name="welcome"></a> + +# Welcome! + +If you are reading this document then you are interested in contributing to the Gearpump project -- many thanks for that! +All contributions are welcome: ideas, documentation, code, patches, bug reports, feature requests, etc. + + +<a name="workflow"></a> +# Contribution workflow + +This section explains how to make a contribution. + + +<a name="report-bug"></a> + +## Report a bug + +To report a bug you should [open an issue](https://issues.apache.org/jira/browse/GEARPUMP) in our issue tracker that +summarizes the bug. Set the form field "Issue type" to "Bug". If you have not used the issue tracker before you will +need to register an account (free), log in, and then click on the red "Create Issue" button in the top navigation bar. + +In order to help us understand and fix the bug it would be great if you could provide us with: + +1. The steps to reproduce the bug. This includes information about e.g. the Gearpump version you are using, the deployment model, etc. +2. The expected behavior. +3. The actual, incorrect behavior. + +Feel free to search the issue tracker for existing issues (aka tickets) that already describe the problem; if there is +such a ticket please add your information as a comment. + +**If you want to provide a patch along with your bug report:** +That is great! In this case please send us a pull request as described in section [Create a pull request](#create-pr) below. +You can also opt to attach a patch file to the issue ticket, but we prefer pull requests because they are easier to work +with. - ``` - git checkout master - git fetch upstream - git rebase upstream/master - ``` -4. Create a working branch +<a name="request-feature"></a> + +## Request a new feature + +To request a new feature you should [open an issue](https://issues.apache.org/jira/browse/GEARPUMP) in our issue tracker +and summarize the desired functionality. Set the form field "Issue type" to "New feature". If you have not used the +issue tracker before you will need to register an account (free), log in, and then click on the red "Create Issue" +button in the top navigation bar. + + +<a name="contribute-code"></a> + +## Contribute code/document by creating Pull Request +Before you set out to contribute code we recommend that you familiarize yourself with the Gearpump codebase and corresponding development document at Gearpump website. + +_If you are interested in contributing code to Gearpump but do not know where to begin:_ +In this case you should +[browse our issue tracker for open issues and tasks](https://issues.apache.org/jira/browse/GEARPUMP/?selectedTab=com.atlassian.jira.jira-projects-plugin:issues-panel). + +Contributions to the Gearpump codebase should be sent as GitHub pull requests. See section [Create a pull request](#create-pr) below +for details. If there is any problem with the pull request we can iterate on it using the commenting features of +GitHub. + +* For _small patches_, feel free to submit pull requests directly for those patches. +* For _larger code contributions_, please use the following process. The idea behind this process is to prevent any + wasted work and catch design issues early on. + + 1. [Open an issue](https://issues.apache.org/jira/browse/GEARPUMP) on our issue tracker if a similar issue does not + exist already. If a similar issue does exist, then you may consider participating in the work on the existing + issue. + 2. Comment on the issue with your plan for implementing the issue. Explain what pieces of the codebase you are + going to touch and how everything is going to fit together. + 3. Gearpump committers will iterate with you on the design to make sure you are on the right track. + 4. Implement your issue, create a pull request (see below), and iterate from there. + + +### Contribution Guideline +1. For all commits, an issue id must be created. +2. For all commit log messages, they must contain issue id. Like this: "fix #issueId, comments". +3. For all PRs (pull request), the title must contains issue Id. +4. We use rebase and squash instead of merge to ensure the log message is clean. Check section "Pull Request merge process for Gearpump" +5. Every commit (except doc) must have 1 committer to review before commit. + +<a name="create-pr"></a> +### Create a Pull Request +Before working on code contribution, you need to prepare your [development environment](#build-and-test). + +To work on a code contribution, following process is suggested: + +1. You need to create a working branch (before that, please make sure your local master is synced with **upstream** master.) ``` git branch branch_issueId git checkout branch_issueId ``` - Work on the branch, make changes, then push to your forked gearpump repo http://github.com/<git-user_id>/gearpump + Work on the branch, make changes, then push to [your forked Gearpump repo](https://github.com/your_username/incubator-gearpump). ``` git push origin branch_issueId ``` -5. When there is changes in upstream/master, rebase your work on upstream/master. with +2. When there are changes in upstream/master, rebase your work on upstream/master. ```bash git checkout branch_issueId + git fetch upstream git rebase -i upstream/master ``` You can also use this when you want to squash(merge) multiple commits into one. - ```git rebase -i``` will popup a window, which allow you to squash(merge) multiple commits into one commit. - For example I might have 12 commits in my branch. "rebase -i upstream/master" opens a nice editor where you can mark some commits to be squashed(merged) into prior commits, and make 1 big commit (or several) out of it. In this way, I can tidy up what will be committed to the project master's history since otherwise my commit messages are like "not working" or "got it working" or "more fix" or "merged <git-user-id>/gearpump to master". + ```git rebase -i``` will pop up a window, which allows you to squash(merge) multiple commits into one commit. + For example I might have 12 commits in my branch. ```git rebase -i upstream/master``` opens a nice editor where you can mark some commits to be squashed(merged) into prior commits, and make 1 big commit (or several) out of it. In this way, I can tidy up what will be committed to the project master's history since otherwise my commit messages are like "not working" or "got it working" or "more fix". -6. If there is conflict, resolve the conflict, and then +3. If there is conflict, resolve the conflict, and then ```bash git rebase --continue ``` After the code is successfully rebased, a window will pop up to edit the commit log, edit it then save and exit. -7. After rebase, now you have a clean log history. push to your remote working branch +4. After rebase, now you have a clean log history. push to your remote working branch ``` git push origin branch_issueId ``` - If commits have already been pushed to <git-user-id>/gearpump fork on github, you will have to "git push --force" to overwrite them with squashed commits. + If commits have already been pushed to your forked repository on GitHub, you will have to "git push --force" to overwrite them with squashed commits. ``` git push origin -f branch_issueId ``` -8. Ensure all the unit tests are passed by running command "sbt test". -6. Open a PR, which is a one-click thing in github.com; it knows you likely are opening a PR against upstream master. [Guide](https://help.github.com/articles/creating-a-pull-request) is here. -7. [Merge PR](https://help.github.com/articles/merging-a-pull-request), [delete branch](https://help.github.com/articles/deleting-unused-branches). +5. Ensure all the unit tests and integration tests are passed, check [Test](#test) for details. +6. Open a Pull Request, which is a one-click thing in github.com; it knows you likely are opening a PR against upstream master. [Guide](https://help.github.com/articles/creating-a-pull-request) is here. + +<a name="code-review"></a> +# Code Review +Committer will review your code periodically. When there is any comment/feedback from committer(s), it's contributor's duty to update the pull request correspondingly. + +When the merge is done by committer, you can optionally [delete your PR branch](https://help.github.com/articles/deleting-unused-branches). + + +<a name="build-and-test"></a> + +# Build and test +Though without a development environment setup, you can still contribute to Gearpump by reporting ideas, documentations, bugs and feature requests. +It is highly recommended to set up a development environment to make any code contribution. + +<a name="local-copy"></a> +## Clone Gearpump repository and make a local copy +If you just want to study Gearpump source code, it is optional to perform following steps. +But, if you plan to contribute to Gearpump's code base, it is necessary to perform following steps: + +1. [Fork](https://help.github.com/articles/fork-a-repo) https://github.com/apache/incubator-gearpump to your own repo at https://github.com/your_username/incubator-gearpump. + +2. Clone the forked repo at your computer. + + ```bash + git clone https://github.com/your_username/incubator-gearpump + cd incubator-gearpump + ``` + +3. Add apache/incubator-gearpump as an external repo 'upstream' by following the [guide](https://help.github.com/articles/configuring-a-remote-for-a-fork/). + + ```bash + git remote add upstream https://github.com/apache/incubator-gearpump.git + ``` + +4. In local master branch, periodically sync the forked master with the upstream master with + + ```bash + git pull --rebase upstream master + git push origin master + ``` + +Another way to do this is to + + ```bash + git checkout master + git fetch upstream + git rebase upstream/master + ``` + +No development work should ever be done in the forked master. + +<a name="build"></a> +## How to build +To make a compilation of Gearpump, you can execute: +```bash + sbt compile pack +``` + +To build a Gearpump package, you can execute following commands: + +```bash + ## The target package path: output/target/gearpump-${version}.zip + sbt clean +assembly +packArchiveZip +``` + + After the build, there will be a package file gearpump-${version}.zip generated under output/target/ folder. + + **NOTE:** +The build requires network connection. If you are behind a proxy, make sure you have set the proxy in your env before running the build commands. + +For Windows: + +```bash +set HTTP_PROXY=http://host:port +set HTTPS_PROXY= http://host:port +``` + +For Linux: + +```bash +export HTTP_PROXY=http://host:port +export HTTPS_PROXY= http://host:port +``` +<a name="test"></a> +## How to test +Unit tests and Integration tests are an essential part of code contributions. + +To run unit test, you can run +```bash + sbt test +``` + +Gearpump has an integration test system which is based on Docker. Please check [the instructions](integrationtest/README.md). + +<a name="build-doc"></a> +## How to build Gearpump documentation +To build Gearpump document use +```bash + ## the 2nd argument indicates whether to build API doc or not + ## for release, we need to use '1' to build it (time consuming) + docs/build_doc.sh 2.11 0 +``` + +<a name="ide-setup"></a> +## IDE setup +IDE environment can be set up on either Windows, Linux and Mac platform. You can choose the one you prefer. +The IDE setup guide can be found at [Gearpump website](http://www.gearpump.io/releases/latest/dev-ide-setup.html). + +It is highly recommended to perform [package build](#build) before IDE setup. + +<a name="code-style"></a> +## Gearpump code style + +Gearpump follows the standard Scala style, just like other Scala projects. This style is: + +1. Mainly, we follow the [Spark Scala style](https://github.com/databricks/scala-style-guide). + +2. We allow some exceptions: e.g. allowing using !, ? for Akka to send actor message. + +Before submitting a PR, you should always run style check first: +``` + ## Run style check for compile, test, and integration test. + sbt scalastyle test:scalastyle it:scalastyle +``` + +<a name="write-unittest"></a> +## How to write unit test +TBD + +<a name="write-integrationtest"></a> +## How to write integration test +TBD + +<a name="write-doc"></a> +## How to write document +Documentation contributions are very welcome! + +You can contribute documentation by pull request, same as code contribution. +Main directory is ```docs/```, and you can refer to docs/README.md for how to build / test documentation. + +NOTE: this documentation is the release documentation for Gearpump, not the website documentation at Apache website, which is maintained by another repository. + + +<a name="committer-work"></a> +# Committer section +_This section applies to committers only._ + +<a name="approve-pull-request"></a> +## Approve pull request +It's committer's duty to review pull requests from contributors. + +Any PR ready to merge shall have at least one +1(s) and no -1(s) from other than the one who authored this PR. And any merge shall wait another 24 hours after the first +1 received to wait for potential comments. +Only committer has the right to perform PR merge to Apache upstream. + + +<a name="merge-pull-request"></a> +## Merge pull request -You can skip using branches in your fork if desired, and just work out of your master. Less overhead, but, you can't pursue different unrelated changes then. +All merges should be done using the `dev-tools/merge_gearpump_pr.py` script. To use this script, you will need to add a git remote called "apache" at `https://git-wip-us.apache.org/repos/asf/incubator-gearpump.git`, as well as one called "apache-github" at `git://github.com/apache/incubator-gearpump`. For the "apache" repo, you can authenticate using your ASF username and password. -The key here is using rebase, not merge. The downstream forks never need to use merge unless the work is a long-lived collaboration with someone else who is pulling from your fork. +The script is fairly self explanatory and walks you through following steps and options interactively. -More details on Squash before pull request ---------------------------- -More about squash, please check +1. squashes the pull request's changes into one commit and merge into master +2. push to "apache" repo (automitically close GitHub pull request) +3. optionally cherry-pick the commit on to another branch +5. clean up and resolve the related JIRA issue -http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html +If you want to amend a commit before merging â which should be used for trivial touch-ups â then simply let the script wait at the point where it asks you if you want to push to Apache. Then, in a separate window, modify the code and push a commit. Run "git rebase -i HEAD~2" and "squash" your new commit. Edit the commit message just after to remove your commit message. You can verify the result is one change with "git log". Then resume the script in the other window. +Also, please remember to set Assignee on JIRAs where applicable when they are resolved. The script can't do this automatically. -https://github.com/edx/edx-platform/wiki/How-to-Rebase-a-Pull-Request +<a name="release"></a> +## How to make a release +Follow [release process](ReleaseProcess.md) for making a release. http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/328063f5/dev-tools/merge_gearpump_pr.py ---------------------------------------------------------------------- diff --git a/dev-tools/merge_gearpump_pr.py b/dev-tools/merge_gearpump_pr.py new file mode 100755 index 0000000..3cbb73f --- /dev/null +++ b/dev-tools/merge_gearpump_pr.py @@ -0,0 +1,378 @@ +#!/usr/bin/env python + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Modifed from the script written by the Apache Spark team: +# https://github.com/apache/spark/blob/master/dev/merge_spark_pr.py + +# Utility for creating well-formed pull request merges and pushing them to Apache. +# usage: ./merge_gearpump_pr.py (see config env vars below) +# +# This utility assumes you already have a local Gearpump git folder and that you +# have added remotes corresponding to both +# (i) the github apache Gearpump mirror +# (ii) the apache git repo. + +import json +import os +import re +import subprocess +import sys +import urllib2 + +try: + import jira.client + JIRA_IMPORTED = True +except ImportError: + JIRA_IMPORTED = False + +# Location of your Gearpump git development area +GEARPUMP_HOME = os.environ.get("GEARPUMP_HOME", os.getcwd()) +# Remote name which points to git://github.com/apache/incubator-gearpump +PR_REMOTE_NAME = os.environ.get("PR_REMOTE_NAME", "apache-github") +# Remote name which points to https://git-wip-us.apache.org/repos/asf/incubator-gearpump.git +PUSH_REMOTE_NAME = os.environ.get("PUSH_REMOTE_NAME", "apache") +# ASF JIRA username +JIRA_USERNAME = os.environ.get("JIRA_USERNAME", "mauzhang") +# ASF JIRA password +JIRA_PASSWORD = os.environ.get("JIRA_PASSWORD", "jira$1024") + + +GITHUB_BASE = "https://github.com/apache/incubator-gearpump/pull"; +GITHUB_API_BASE = "https://api.github.com/repos/apache/incubator-gearpump"; +JIRA_BASE = "https://issues.apache.org/jira/browse"; +JIRA_API_BASE = "https://issues.apache.org/jira"; +# Prefix added to temporary branches +BRANCH_PREFIX = "PR_TOOL" + + +def get_json(url): + try: + return json.load(urllib2.urlopen(url)) + except urllib2.HTTPError as e: + print "Unable to fetch URL, exiting: %s" % url + sys.exit(-1) + + +def fail(msg): + print msg + clean_up() + sys.exit(-1) + + +def run_cmd(cmd): + print cmd + if isinstance(cmd, list): + return subprocess.check_output(cmd) + else: + return subprocess.check_output(cmd.split(" ")) + + +def continue_maybe(prompt): + result = raw_input("\n%s (y/n): " % prompt) + if result.lower() != "y": + fail("Okay, exiting") + +def clean_up(): + print "Restoring head pointer to %s" % original_head + run_cmd("git checkout %s" % original_head) + + branches = run_cmd("git branch").replace(" ", "").split("\n") + + for branch in filter(lambda x: x.startswith(BRANCH_PREFIX), branches): + print "Deleting local branch %s" % branch + run_cmd("git branch -D %s" % branch) + + +# merge the requested PR and return the merge hash +def merge_pr(pr_num, target_ref, title, body, pr_repo_desc): + pr_branch_name = "%s_MERGE_PR_%s" % (BRANCH_PREFIX, pr_num) + target_branch_name = "%s_MERGE_PR_%s_%s" % (BRANCH_PREFIX, pr_num, target_ref.upper()) + run_cmd("git fetch %s pull/%s/head:%s" % (PR_REMOTE_NAME, pr_num, pr_branch_name)) + run_cmd("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, target_ref, target_branch_name)) + run_cmd("git checkout %s" % target_branch_name) + + had_conflicts = False + try: + run_cmd(['git', 'merge', pr_branch_name, '--squash']) + except Exception as e: + msg = "Error merging: %s\nWould you like to manually fix-up this merge?" % e + continue_maybe(msg) + msg = "Okay, please fix any conflicts and 'git add' conflicting files... Finished?" + continue_maybe(msg) + had_conflicts = True + + commit_authors = run_cmd(['git', 'log', 'HEAD..%s' % pr_branch_name, + '--pretty=format:%an <%ae>']).split("\n") + distinct_authors = sorted(set(commit_authors), + key=lambda x: commit_authors.count(x), reverse=True) + + primary_author = distinct_authors[0] + + commits = run_cmd(['git', 'log', 'HEAD..%s' % pr_branch_name, + '--pretty=format:%h [%an] %s']).split("\n\n") + + merge_message_flags = [] + + merge_message_flags += ["-m", title] + if body is not None: + # We remove @ symbols from the body to avoid triggering e-mails + # to people every time someone creates a public fork of Gearpump. + merge_message_flags += ["-m", body.replace("@", "")] + + authors = "\n".join(["Author: %s" % a for a in distinct_authors]) + + merge_message_flags += ["-m", authors] + + if had_conflicts: + committer_name = run_cmd("git config --get user.name").strip() + committer_email = run_cmd("git config --get user.email").strip() + message = "This patch had conflicts when merged, resolved by\nCommitter: %s <%s>" % ( + committer_name, committer_email) + merge_message_flags += ["-m", message] + + # The string "Closes #%s" string is required for GitHub to correctly close the PR + merge_message_flags += ["-m", "Closes #%s from %s." % (pr_num, pr_repo_desc)] + + run_cmd(['git', 'commit', '--author="%s"' % primary_author] + merge_message_flags) + + continue_maybe("Merge complete (local ref %s). Push to %s?" % ( + target_branch_name, PUSH_REMOTE_NAME)) + + try: + run_cmd('git push %s %s:%s' % (PUSH_REMOTE_NAME, target_branch_name, target_ref)) + except Exception as e: + clean_up() + fail("Exception while pushing: %s" % e) + + merge_hash = run_cmd("git rev-parse %s" % target_branch_name)[:8] + clean_up() + print("Pull request #%s merged!" % pr_num) + print("Merge hash: %s" % merge_hash) + return merge_hash + + +def cherry_pick(pr_num, merge_hash, default_branch): + pick_ref = raw_input("Enter a branch name [%s]: " % default_branch) + if pick_ref == "": + pick_ref = default_branch + + pick_branch_name = "%s_PICK_PR_%s_%s" % (BRANCH_PREFIX, pr_num, pick_ref.upper()) + + run_cmd("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, pick_ref, pick_branch_name)) + run_cmd("git checkout %s" % pick_branch_name) + + try: + run_cmd("git cherry-pick -sx %s" % merge_hash) + except Exception as e: + msg = "Error cherry-picking: %s\nWould you like to manually fix-up this merge?" % e + continue_maybe(msg) + msg = "Okay, please fix any conflicts and finish the cherry-pick. Finished?" + continue_maybe(msg) + + continue_maybe("Pick complete (local ref %s). Push to %s?" % ( + pick_branch_name, PUSH_REMOTE_NAME)) + + try: + run_cmd('git push %s %s:%s' % (PUSH_REMOTE_NAME, pick_branch_name, pick_ref)) + except Exception as e: + clean_up() + fail("Exception while pushing: %s" % e) + + pick_hash = run_cmd("git rev-parse %s" % pick_branch_name)[:8] + clean_up() + + print("Pull request #%s picked into %s!" % (pr_num, pick_ref)) + print("Pick hash: %s" % pick_hash) + return pick_ref + + +def fix_version_from_branch(branch, versions): + # Note: Assumes this is a sorted (newest->oldest) list of un-released versions + if branch == "master": + return versions[0] + else: + branch_ver = branch.replace("branch-", "") + return filter(lambda x: x.name.startswith(branch_ver), versions)[-1] + + +def resolve_jira_issue(merge_branches, comment, default_jira_id=""): + asf_jira = jira.client.JIRA({'server': JIRA_API_BASE}, + basic_auth=(JIRA_USERNAME, JIRA_PASSWORD)) + + jira_id = raw_input("Enter a JIRA id [%s]: " % default_jira_id) + if jira_id == "": + jira_id = default_jira_id + + try: + issue = asf_jira.issue(jira_id) + except Exception as e: + fail("ASF JIRA could not find %s\n%s" % (jira_id, e)) + + cur_status = issue.fields.status.name + cur_summary = issue.fields.summary + cur_assignee = issue.fields.assignee + if cur_assignee is None: + cur_assignee = "NOT ASSIGNED!!!" + else: + cur_assignee = cur_assignee.displayName + + if cur_status == "Resolved" or cur_status == "Closed": + fail("JIRA issue %s already has status '%s'" % (jira_id, cur_status)) + print ("=== JIRA %s ===" % jira_id) + print ("summary\t\t%s\nassignee\t%s\nstatus\t\t%s\nurl\t\t%s/%s\n" % ( + cur_summary, cur_assignee, cur_status, JIRA_BASE, jira_id)) + + versions = asf_jira.project_versions("GEARPUMP") + versions = sorted(versions, key=lambda x: x.name, reverse=True) + versions = filter(lambda x: x.raw['released'] is False, versions) + # Consider only x.y.z versions + versions = filter(lambda x: re.match('\d+\.\d+\.\d+', x.name), versions) + + default_fix_versions = map(lambda x: fix_version_from_branch(x, versions).name, merge_branches) + for v in default_fix_versions: + # Handles the case where we have forked a release branch but not yet made the release. + # In this case, if the PR is committed to the master branch and the release branch, we + # only consider the release branch to be the fix version. E.g. it is not valid to have + # both 1.1.0 and 1.0.0 as fix versions. + (major, minor, patch) = v.split(".") + if patch == "0": + previous = "%s.%s.%s" % (major, int(minor) - 1, 0) + if previous in default_fix_versions: + default_fix_versions = filter(lambda x: x != v, default_fix_versions) + default_fix_versions = ",".join(default_fix_versions) + + fix_versions = raw_input("Enter comma-separated fix version(s) [%s]: " % default_fix_versions) + if fix_versions == "": + fix_versions = default_fix_versions + fix_versions = fix_versions.replace(" ", "").split(",") + + def get_version_json(version_str): + return filter(lambda v: v.name == version_str, versions)[0].raw + + jira_fix_versions = map(lambda v: get_version_json(v), fix_versions) + + resolve = filter(lambda a: a['name'] == "Resolve Issue", asf_jira.transitions(jira_id))[0] + resolution = filter(lambda r: r.raw['name'] == "Fixed", asf_jira.resolutions())[0] + asf_jira.transition_issue( + jira_id, resolve["id"], fixVersions = jira_fix_versions, + comment = comment, resolution = {'id': resolution.raw['id']}) + + print "Successfully resolved %s with fixVersions=%s!" % (jira_id, fix_versions) + + +def resolve_jira_issues(title, merge_branches, comment): + jira_ids = re.findall("GEARPUMP-[0-9]+", title) + + if len(jira_ids) == 0: + resolve_jira_issue(merge_branches, comment) + for jira_id in jira_ids: + resolve_jira_issue(merge_branches, comment, jira_id) + +def get_current_ref(): + ref = run_cmd("git rev-parse --abbrev-ref HEAD").strip() + if ref == 'HEAD': + # The current ref is a detached HEAD, so grab its SHA. + return run_cmd("git rev-parse HEAD").strip() + else: + return ref + + +def main(): + global original_head + + os.chdir(GEARPUMP_HOME) + original_head = get_current_ref() + + branches = get_json("%s/branches" % GITHUB_API_BASE) + branch_names = filter(lambda x: True, [x['name'] for x in branches]) + # Assumes branch names can be sorted lexicographically + latest_branch = sorted(branch_names, reverse=True)[0] + + pr_num = raw_input("Which pull request would you like to merge? (e.g. 34): ") + pr = get_json("%s/pulls/%s" % (GITHUB_API_BASE, pr_num)) + pr_events = get_json("%s/issues/%s/events" % (GITHUB_API_BASE, pr_num)) + + url = pr["url"] + + title = pr["title"] + body = pr["body"] + target_ref = pr["base"]["ref"] + user_login = pr["user"]["login"] + base_ref = pr["head"]["ref"] + pr_repo_desc = "%s/%s" % (user_login, base_ref) + + # Merged pull requests don't appear as merged in the GitHub API; + # Instead, they're closed by asfgit. + merge_commits = \ + [e for e in pr_events if e["actor"]["login"] == "asfgit" and e["event"] == "closed"] + + if merge_commits: + merge_hash = merge_commits[0]["commit_id"] + message = get_json("%s/commits/%s" % (GITHUB_API_BASE, merge_hash))["commit"]["message"] + + print "Pull request %s has already been merged, assuming you want to backport" % pr_num + commit_is_downloaded = run_cmd(['git', 'rev-parse', '--quiet', '--verify', + "%s^{commit}" % merge_hash]).strip() != "" + if not commit_is_downloaded: + fail("Couldn't find any merge commit for #%s, you may need to update HEAD." % pr_num) + + print "Found commit %s:\n%s" % (merge_hash, message) + cherry_pick(pr_num, merge_hash, latest_branch) + sys.exit(0) + + if not bool(pr["mergeable"]): + msg = "Pull request %s is not mergeable in its current form.\n" % pr_num + \ + "Continue? (experts only!)" + continue_maybe(msg) + + print ("\n=== Pull Request #%s ===" % pr_num) + print ("title\t%s\nsource\t%s\ntarget\t%s\nurl\t%s" % ( + title, pr_repo_desc, target_ref, url)) + continue_maybe("Proceed with merging pull request #%s?" % pr_num) + + merged_refs = [target_ref] + + merge_hash = merge_pr(pr_num, target_ref, title, body, pr_repo_desc) + + pick_prompt = "Would you like to pick %s into another branch?" % merge_hash + while raw_input("\n%s (y/n): " % pick_prompt).lower() == "y": + merged_refs = merged_refs + [cherry_pick(pr_num, merge_hash, latest_branch)] + + if JIRA_IMPORTED: + if JIRA_USERNAME and JIRA_PASSWORD: + continue_maybe("Would you like to update an associated JIRA?") + jira_comment = "Issue resolved by pull request %s\n[%s/%s]" % (pr_num, GITHUB_BASE, pr_num) + resolve_jira_issues(title, merged_refs, jira_comment) + else: + print "JIRA_USERNAME and JIRA_PASSWORD not set" + print "Exiting without trying to close the associated JIRA." + else: + print "Could not find jira-python library. Run 'sudo pip install jira' to install." + print "Exiting without trying to close the associated JIRA." + +if __name__ == "__main__": + import doctest + (failure_count, test_count) = doctest.testmod() + if failure_count: + exit(-1) + try: + main() + except: + clean_up() + raise
