There's a lot more in dev-support than just the release tooling; we can't delete the entirety of that folder from all branches currently. Each branch needs to have several Jenkinsfiles for our jobs that use the Jenkins Multibranch Pipeline for our various automated testing (I think there are ~4 of them). Each branch also needs to have the dockerfile needed for Yetus to run nightly and precommit against that branch.
Even with those things left in place it'll take a non-trivial amount of effort to get the nightly jobs on all branches to work off of support scripts out of the master branch. It's not clear to me that anyone thus far in this thread has volunteered to do that work. On Sat, Sep 28, 2019 at 6:06 PM Stack <[email protected]> wrote: > > On Sat, Sep 28, 2019 at 11:13 AM Nick Dimiduk <[email protected]> wrote: > > > On Sat, Sep 28, 2019 at 10:58 Stack <[email protected]> wrote: > > > > > hbase-operator-tools/create-release or a new repo altogether? > > > > > > > If we’re going to have one tool to release them all, I’m okay with it > > staying in the main repo under dev-support or similar, so long as it’s on > > only one branch (probably master, maybe it’s own). If the tools is itself > > is going to be released and made installable (on a developer laptop or a > > Jenkins worker) then I think it should have its own repo. > > > > > Sounds good. Lets start with master branch of core for now. Can do a > dedicated repo later. I can delete the dev-tools from older branches (and > from other repos where it exists) so only the one instance in one location > going forward. > > S > > > > > > On Wed, Sep 25, 2019 at 8:56 AM Peter Somogyi <[email protected]> > > wrote: > > > > > > > > > > It is great that you brought up this topic Nick! I agree that the > > > optimal > > > > > solution would be to host all the release related scripts (RC build, > > > > > verifier) in a common place. > > > > > > > > > > The RC making scrip in hbase-operator-tools that Stack finetuned is > > > meant > > > > > to work with different artifacts. The current version there gives the > > > RM > > > > a > > > > > smooth experience. It emerged from HBase's create-release script and > > > > > hopefully it can be used on other releases as well. There are some > > > > > constraints of the tool like Jira versions should use > > > > > `hbase-operator-tools` prefix. > > > > > > > > > > > This is a good point. I wonder if `dev-support` should be pruned or > > > > purged > > > > > from all branches other than master > > > > > > > > > > When we create branch-3 out of master then this becomes a problem > > > again. > > > > > Would it work if we use a specific branch for such tooling (e.g. > > > > > dev-tools)? In this case RMs can just clone that branch and don't > > need > > > > the > > > > > whole HBase repository on their local machine. > > > > > > > > > > On Thu, Sep 19, 2019 at 5:40 PM Nick Dimiduk <[email protected]> > > > > wrote: > > > > > > > > > > > On Wed, Sep 18, 2019 at 10:30 AM Sean Busbey <[email protected]> > > > > wrote: > > > > > > > > > > > > > If the tooling is in one place where will it live? > > > > > > > > > > > > > > As an RM I like not needing to checkout more then the repo I'm > > > > trying to > > > > > > > get a release out on. > > > > > > > > > > > > > > > > > > > My initial thinking was all would be in the main repo, but this is > > > > contrary > > > > > > to your above statement. As I see it, everyone working on HBase > > has a > > > > > > checkout of the main repo handy, so for releases spun on developer > > > > machines > > > > > > it's "no big deal." If we can ever get to releases spun through > > > > automated > > > > > > environments, it's all scripted anyway and thus "no big deal." > > > > > > > > > > > > In particular my release machine is slow on disk and so updates to > > > the > > > > main > > > > > > > git repo when trying to release something not in the main repo > > will > > > > be > > > > > > > painful. > > > > > > > > > > > > > > > > > > "slow on disk" as in iops rate or "low on disk" as in capacity? > > > Either > > > > way, > > > > > > I'm surprised to hear about this as a barrier. Is there a > > > > side-conversation > > > > > > we can have about trimming the fat out of our repo(s)? Some git > > > > maintenance > > > > > > that can/needs to be done? I was recently shocked by the girth of > > our > > > > repos > > > > > > -- nearly 0.5g on the main repo and a whopping 4g on the site! > > > > > > > > > > > > For the most part this is also why I usually manually build RCs > > > > > > > that I run for the main repo, because I can do a shallow clone of > > > the > > > > > > > release branch instead of needing to get updates to all the > > active > > > > > > > branches. > > > > > > > > > > > > > > > > > > > This makes me sad. Automate more, not less! Altering the automation > > > to > > > > make > > > > > > shallow clones of targeted branches should be simple enough, no? > > > > > > > > > > > > For testing RCs I guess it's currently all some combination of the > > > > > > > foundation policies that should be the same and maybe maven > > > profiles? > > > > > > > > > > > > > > > > > > > I agree that codification of foundation policy is the baseline. > > That > > > > would > > > > > > be enough for me as a first pass. After that, perhaps layers of > > > > increasing > > > > > > sophistication, perhaps repo-dependent. I don't follow your meaning > > > re: > > > > > > maven profiles. > > > > > > > > > > > > Iirc there was already some confusion about using the testing > > script > > > > that > > > > > > > came in the main repo source bundle vs the one on master. > > > > > > > > > > > > > > > > > > > This is a good point. I wonder if `dev-support` should be pruned or > > > > purged > > > > > > from all branches other than master. Maybe the CI related stuff > > > > branches > > > > > > into it's own directory or directories, and we keep everything else > > > > limited > > > > > > to a single canonical copy on master. This strategy would eliminate > > > > > > confusion re: what is authority in any given situation but perhaps > > is > > > > too > > > > > > constraining, given the number of active release branches we > > > maintain. > > > > > > > > > > > > What issue are we trying to solve? > > > > > > > > > > > > > > > > > > > Minimize contributor friction. Make it easy for every subscriber to > > > > dev@ > > > > > > to > > > > > > say "There's a new RC posted and I have an idle machine for an > > hour / > > > > for > > > > > > the evening / for the weekend, let's just kick the tires." Make it > > > > easy for > > > > > > everyone who's learned the RC tooling on one branch to pinch-hit in > > > on > > > > > > another branch or another repo. I hear a constant complaint of > > > > "scarcity of > > > > > > volunteer hours" and "I wish we had more people voting in RCs" and > > "I > > > > wish > > > > > > we could keep a monthly release cadence on every branch we're > > > > maintaining". > > > > > > So I'd like to see a focused effort on maximizing the volunteer > > human > > > > and > > > > > > machine time that's thrown our way. > > > > > > > > > > > > Thanks, > > > > > > Nick > > > > > > > > > > > > On Wed, Sep 18, 2019, 11:50 Nick Dimiduk <[email protected]> > > > wrote: > > > > > > > > > > > > > > > Heya, > > > > > > > > > > > > > > > > Looks like we have quite a few repos now, each of which must > > > > produce > > > > > > > > artifacts that follow the Apache protocols. I also see we have > > > some > > > > > > nice > > > > > > > > tools built up in dev-support in the main repo for RM's who > > build > > > > > > release > > > > > > > > candidates and community members to vote on them. I tried to > > use > > > > our > > > > > > > > hbase-vote.sh on this operator-tools RC and found it mostly > > > works. > > > > I > > > > > > > think > > > > > > > > a few adjustments on each end would allow these tools to work > > > > across > > > > > > > repos, > > > > > > > > so I filed HBASE-23048. However, I now see that operator-tools > > > has > > > > its > > > > > > > own > > > > > > > > dev-support/create-release, so I wonder what direction we want > > to > > > > take > > > > > > > with > > > > > > > > this automation, so here I come to the list. > > > > > > > > > > > > > > > > Do we want to have independent tooling for each repo? Are the > > > > processes > > > > > > > of > > > > > > > > building RC's different enough to warrant separate tools? Is a > > > > single > > > > > > > tool > > > > > > > > that can build an RC for all of them not worth the trouble? At > > > the > > > > very > > > > > > > > least, I think the hbase-vote.sh can be made to work with > > > releases > > > > > > > > generated from each of the repos, as it's not doing all that > > > much. > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Nick > > > > > > > > > > > > > > > > On Wed, Sep 18, 2019 at 9:42 AM Nick Dimiduk < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > - Verified signatures. > > > > > > > > > - Verified checksums. > > > > > > > > > - Built from source tarball successfully. > > > > > > > > > - Ran unit tests from source tarball, pass. > > > > > > > > > - Ran rat check from source tarball, pass. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 16, 2019 at 11:31 AM Peter Somogyi < > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Please vote on this Apache HBase Operator Tools Release > > > > Candidate > > > > > > > (RC), > > > > > > > > >> 1.0.0. > > > > > > > > >> > > > > > > > > >> The VOTE will remain open for at least 72 hours. > > > > > > > > >> > > > > > > > > >> [ ] +1 Release this package as Apache HBase Operator Tools > > > 1.0.0 > > > > > > > > >> [ ] -1 Do not release this package because ... > > > > > > > > >> > > > > > > > > >> The tag to be voted on is 1.0.0RC0: > > > > > > > > >> > > > > > > > > >> > > https://github.com/apache/hbase-operator-tools/tree/1.0.0RC0 > > > > > > > > >> > > > > > > > > >> The release files, including signatures, digests, as well as > > > > > > > CHANGES.md > > > > > > > > >> and RELEASENOTES.md included in this RC can be found at: > > > > > > > > >> > > > > > > > > >> https://dist.apache.org/repos/dist/dev/hbase/1.0.0RC0/ > > > > > > > > >> > > > > > > > > >> Maven artifacts are available in a staging repository at: > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1348/ > > > > > > > > >> > > > > > > > > >> Artifacts were signed with the [email protected] key > > which > > > > can be > > > > > > > > found > > > > > > > > >> in: > > > > > > > > >> > > > > > > > > >> https://dist.apache.org/repos/dist/release/hbase/KEYS > > > > > > > > >> > > > > > > > > >> To learn more about Apache HBase Operator Tools, please see > > > > > > > > >> http://hbase.apache.org/ > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> Your HBase Release Manager > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
