dev-support is definitely suitable for breaking out. It rarely changes. When it does we have issues keeping the bits in sync across all the branches in the main repo. +1
That said the proliferation of repos is a concern. The last time I went looking to evaluate HBase 2 I was faced by a bewildering sprawl of repos and branches while trying to assemble the complete solution. Main repo? Third party? Connectors? Filesystem? Operator tools? Which in what order and how? A prerequisite for calling a break out of dev-support done will be some clear instructions on end to end build. One page. One list of all the bits needed from all the places and one set of concise build and release instructions. > On Sep 25, 2019, at 7:24 AM, Sean Busbey <[email protected]> wrote: > > I recently spun an RC out of the hbase-thirdparty repo and it changed > my mind from neutral to positive on the idea of consolidating our > release handling stuff. Some of the instructions in that repo have > suffered from bit rot (I think because they relied on running some > steps out of the main repo). > > What if we kept the release management / dev-support stuff in a > dedicated repo? presumably we'd never actually release anything out of > this repo, but if we used a branch in the main repo we wouldn't > release out of it either? > >> On Wed, Sep 25, 2019 at 8:56 AM Peter Somogyi <[email protected]> wrote: >> >> It is great that you brought up this topic Nick! I agree that the optimal >> solution would be to host all the release related scripts (RC build, >> verifier) in a common place. >> >> The RC making scrip in hbase-operator-tools that Stack finetuned is meant >> to work with different artifacts. The current version there gives the RM a >> smooth experience. It emerged from HBase's create-release script and >> hopefully it can be used on other releases as well. There are some >> constraints of the tool like Jira versions should use >> `hbase-operator-tools` prefix. >> >>> This is a good point. I wonder if `dev-support` should be pruned or purged >> from all branches other than master >> >> When we create branch-3 out of master then this becomes a problem again. >> Would it work if we use a specific branch for such tooling (e.g. >> dev-tools)? In this case RMs can just clone that branch and don't need the >> whole HBase repository on their local machine. >> >>> On Thu, Sep 19, 2019 at 5:40 PM Nick Dimiduk <[email protected]> wrote: >>> >>>> On Wed, Sep 18, 2019 at 10:30 AM Sean Busbey <[email protected]> wrote: >>>> >>>> If the tooling is in one place where will it live? >>>> >>>> As an RM I like not needing to checkout more then the repo I'm trying to >>>> get a release out on. >>>> >>> >>> My initial thinking was all would be in the main repo, but this is contrary >>> to your above statement. As I see it, everyone working on HBase has a >>> checkout of the main repo handy, so for releases spun on developer machines >>> it's "no big deal." If we can ever get to releases spun through automated >>> environments, it's all scripted anyway and thus "no big deal." >>> >>> In particular my release machine is slow on disk and so updates to the main >>>> git repo when trying to release something not in the main repo will be >>>> painful. >>> >>> >>> "slow on disk" as in iops rate or "low on disk" as in capacity? Either way, >>> I'm surprised to hear about this as a barrier. Is there a side-conversation >>> we can have about trimming the fat out of our repo(s)? Some git maintenance >>> that can/needs to be done? I was recently shocked by the girth of our repos >>> -- nearly 0.5g on the main repo and a whopping 4g on the site! >>> >>> For the most part this is also why I usually manually build RCs >>>> that I run for the main repo, because I can do a shallow clone of the >>>> release branch instead of needing to get updates to all the active >>>> branches. >>>> >>> >>> This makes me sad. Automate more, not less! Altering the automation to make >>> shallow clones of targeted branches should be simple enough, no? >>> >>> For testing RCs I guess it's currently all some combination of the >>>> foundation policies that should be the same and maybe maven profiles? >>>> >>> >>> I agree that codification of foundation policy is the baseline. That would >>> be enough for me as a first pass. After that, perhaps layers of increasing >>> sophistication, perhaps repo-dependent. I don't follow your meaning re: >>> maven profiles. >>> >>> Iirc there was already some confusion about using the testing script that >>>> came in the main repo source bundle vs the one on master. >>>> >>> >>> This is a good point. I wonder if `dev-support` should be pruned or purged >>> from all branches other than master. Maybe the CI related stuff branches >>> into it's own directory or directories, and we keep everything else limited >>> to a single canonical copy on master. This strategy would eliminate >>> confusion re: what is authority in any given situation but perhaps is too >>> constraining, given the number of active release branches we maintain. >>> >>> What issue are we trying to solve? >>>> >>> >>> Minimize contributor friction. Make it easy for every subscriber to dev@ >>> to >>> say "There's a new RC posted and I have an idle machine for an hour / for >>> the evening / for the weekend, let's just kick the tires." Make it easy for >>> everyone who's learned the RC tooling on one branch to pinch-hit in on >>> another branch or another repo. I hear a constant complaint of "scarcity of >>> volunteer hours" and "I wish we had more people voting in RCs" and "I wish >>> we could keep a monthly release cadence on every branch we're maintaining". >>> So I'd like to see a focused effort on maximizing the volunteer human and >>> machine time that's thrown our way. >>> >>> Thanks, >>> Nick >>> >>>> On Wed, Sep 18, 2019, 11:50 Nick Dimiduk <[email protected]> wrote: >>>> >>>>> Heya, >>>>> >>>>> Looks like we have quite a few repos now, each of which must produce >>>>> artifacts that follow the Apache protocols. I also see we have some >>> nice >>>>> tools built up in dev-support in the main repo for RM's who build >>> release >>>>> candidates and community members to vote on them. I tried to use our >>>>> hbase-vote.sh on this operator-tools RC and found it mostly works. I >>>> think >>>>> a few adjustments on each end would allow these tools to work across >>>> repos, >>>>> so I filed HBASE-23048. However, I now see that operator-tools has its >>>> own >>>>> dev-support/create-release, so I wonder what direction we want to take >>>> with >>>>> this automation, so here I come to the list. >>>>> >>>>> Do we want to have independent tooling for each repo? Are the processes >>>> of >>>>> building RC's different enough to warrant separate tools? Is a single >>>> tool >>>>> that can build an RC for all of them not worth the trouble? At the very >>>>> least, I think the hbase-vote.sh can be made to work with releases >>>>> generated from each of the repos, as it's not doing all that much. >>>>> >>>>> Thoughts? >>>>> >>>>> Thanks, >>>>> Nick >>>>> >>>>> On Wed, Sep 18, 2019 at 9:42 AM Nick Dimiduk <[email protected]> >>>> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> - Verified signatures. >>>>>> - Verified checksums. >>>>>> - Built from source tarball successfully. >>>>>> - Ran unit tests from source tarball, pass. >>>>>> - Ran rat check from source tarball, pass. >>>>>> >>>>>> >>>>>> On Mon, Sep 16, 2019 at 11:31 AM Peter Somogyi <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Please vote on this Apache HBase Operator Tools Release Candidate >>>> (RC), >>>>>>> 1.0.0. >>>>>>> >>>>>>> The VOTE will remain open for at least 72 hours. >>>>>>> >>>>>>> [ ] +1 Release this package as Apache HBase Operator Tools 1.0.0 >>>>>>> [ ] -1 Do not release this package because ... >>>>>>> >>>>>>> The tag to be voted on is 1.0.0RC0: >>>>>>> >>>>>>> https://github.com/apache/hbase-operator-tools/tree/1.0.0RC0 >>>>>>> >>>>>>> The release files, including signatures, digests, as well as >>>> CHANGES.md >>>>>>> and RELEASENOTES.md included in this RC can be found at: >>>>>>> >>>>>>> https://dist.apache.org/repos/dist/dev/hbase/1.0.0RC0/ >>>>>>> >>>>>>> Maven artifacts are available in a staging repository at: >>>>>>> >>>>>>> >>>>> >>> https://repository.apache.org/content/repositories/orgapachehbase-1348/ >>>>>>> >>>>>>> Artifacts were signed with the [email protected] key which can be >>>>> found >>>>>>> in: >>>>>>> >>>>>>> https://dist.apache.org/repos/dist/release/hbase/KEYS >>>>>>> >>>>>>> To learn more about Apache HBase Operator Tools, please see >>>>>>> http://hbase.apache.org/ >>>>>>> >>>>>>> Thanks, >>>>>>> Your HBase Release Manager >>>>>>> >>>>>> >>>>> >>>> >>>
