I'd like to propose that we migrate the ZooKeeper pre-commit process to use Apache Yetus [1].
Apache Yetus provides a set of tools for managing patch contributions and project releases. One component of that is a pre-commit script that offers a lot of enhanced functionality over the current ZooKeeper test-patch.sh. It began life as a rewrite of Hadoop's test-patch.sh. It looks to me like the ZooKeeper test-patch.sh is a fork of an earlier version of Hadoop's test-patch.sh. That's a common pattern throughout Hadoop ecosystem projects. One of the goals of Yetus is to consolidate all of those forks and centralize, so that all projects share the benefits of enhancements. You can see examples of Yetus in action on Hadoop JIRA issues [2]. Here are a few of the features of Yetus pre-commit that I really appreciate: - Run pre-commit against any branch, not just trunk. You can embed the branch name in the patch file name to tell Yetus to apply it to that specific branch. No more explaining to contributors that pre-commit is only for trunk, and no need to run manual tests ourselves for branch-3.4 maintenance patches. - Support applying patches from GitHub pull requests in addition to the traditional JIRA attachment technique. - Detect file types included in the patch, and tune pre-commit checks accordingly. For example, detect that a patch only contains documentation files, so only check "ant docs" and save time by skipping the unit tests. Get more rapid feedback focused on what is important for each particular patch. - Run the build against multiple different JDK versions. - Nicely formatted reporting in JIRA comments. - Flexible plugin architecture based on composing shell scripts. Already includes plugins for many standard checks, such as RAT, Checkstyle, Findbugs and ShellCheck. New checks can be implemented as new plugins by writing another shell script. (Patches welcome!) - Supports customization. We can opt in to what makes sense for ZooKeeper and opt out of what doesn't make sense. If we decide to proceed, the migration process would look like this: 1. Write a pre-commit personality for ZooKeeper. This is a small shell script checked in to the Yetus repo that customizes the build for ZooKeeper's requirements. I have taken the liberty of filing issue YETUS-234 [3] to track this, and I have posted an initial prototype. Since I'm a committer on both ZooKeeper and Yetus, I'll be well-equipped to help us maintain the personality. 2. Test against real ZooKeeper patches. This can be done by creating a bogus ZooKeeper JIRA, attaching patches and triggering a Jenkins job that calls Yetus test-patch.sh with the ZooKeeper personality. This won't impact the main development flow for patches. The only side effect is a bit of extra spam on the mailing list from the testing. 3. Switch the PreCommit-ZOOKEEPER-Build job to call Yetus test-patch.sh instead of the test-patch.sh in the ZooKeeper repository. At this point, it goes live for all patches. If we see unexpected problems, we can switch back to classic test-patch.sh while we investigate. 4. After we're happy with the migration, we can delete test-patch.sh from the ZooKeeper repository. Please let me know your thoughts. If feedback is positive, then I'll proceed with the next steps. Thank you! [1] http://yetus.apache.org/ [2] https://issues.apache.org/jira/browse/HADOOP-12600 [3] https://issues.apache.org/jira/browse/YETUS-234 --Chris Nauroth
