Thanks for proposing this, Chris. I really like the proposal, in particular the ability of testing a patch against different branches.We'll need to change the "How to contribute" document to explain how upload patches and such.
I'm not suggesting we do this here, but since we are talking about improvements, perhaps someone gets interested in pursuing the following: - We should move away from ant and start using either maven or gradle. There is a patch (ZOOKEEPER-1078) for maven by phunt, but hasn't been able to finish it. I'm very supportive of making this move. - I'd rather be using git at this point, so someone would need to the process of requesting infra and such to make the change. -Flavio > On 12 Dec 2015, at 23:18, Chris Nauroth <[email protected]> wrote: > > I'd like to propose that we migrate the ZooKeeper pre-commit process to use > Apache Yetus [1]. > > Apache Yetus provides a set of tools for managing patch contributions and > project releases. One component of that is a pre-commit script that offers a > lot of enhanced functionality over the current ZooKeeper test-patch.sh. It > began life as a rewrite of Hadoop's test-patch.sh. It looks to me like the > ZooKeeper test-patch.sh is a fork of an earlier version of Hadoop's > test-patch.sh. That's a common pattern throughout Hadoop ecosystem projects. > One of the goals of Yetus is to consolidate all of those forks and > centralize, so that all projects share the benefits of enhancements. > > You can see examples of Yetus in action on Hadoop JIRA issues [2]. Here are > a few of the features of Yetus pre-commit that I really appreciate: > > - Run pre-commit against any branch, not just trunk. You can embed the > branch name in the patch file name to tell Yetus to apply it to that specific > branch. No more explaining to contributors that pre-commit is only for > trunk, and no need to run manual tests ourselves for branch-3.4 maintenance > patches. > - Support applying patches from GitHub pull requests in addition to the > traditional JIRA attachment technique. > - Detect file types included in the patch, and tune pre-commit checks > accordingly. For example, detect that a patch only contains documentation > files, so only check "ant docs" and save time by skipping the unit tests. > Get more rapid feedback focused on what is important for each particular > patch. > - Run the build against multiple different JDK versions. > - Nicely formatted reporting in JIRA comments. > - Flexible plugin architecture based on composing shell scripts. Already > includes plugins for many standard checks, such as RAT, Checkstyle, Findbugs > and ShellCheck. New checks can be implemented as new plugins by writing > another shell script. (Patches welcome!) > - Supports customization. We can opt in to what makes sense for ZooKeeper > and opt out of what doesn't make sense. > > If we decide to proceed, the migration process would look like this: > > 1. Write a pre-commit personality for ZooKeeper. This is a small shell > script checked in to the Yetus repo that customizes the build for ZooKeeper's > requirements. I have taken the liberty of filing issue YETUS-234 [3] to > track this, and I have posted an initial prototype. Since I'm a committer on > both ZooKeeper and Yetus, I'll be well-equipped to help us maintain the > personality. > 2. Test against real ZooKeeper patches. This can be done by creating a bogus > ZooKeeper JIRA, attaching patches and triggering a Jenkins job that calls > Yetus test-patch.sh with the ZooKeeper personality. This won't impact the > main development flow for patches. The only side effect is a bit of extra > spam on the mailing list from the testing. > 3. Switch the PreCommit-ZOOKEEPER-Build job to call Yetus test-patch.sh > instead of the test-patch.sh in the ZooKeeper repository. At this point, it > goes live for all patches. If we see unexpected problems, we can switch back > to classic test-patch.sh while we investigate. > 4. After we're happy with the migration, we can delete test-patch.sh from the > ZooKeeper repository. > > Please let me know your thoughts. If feedback is positive, then I'll proceed > with the next steps. Thank you! > > [1] http://yetus.apache.org/ > [2] https://issues.apache.org/jira/browse/HADOOP-12600 > [3] https://issues.apache.org/jira/browse/YETUS-234 > > --Chris Nauroth
