Thanks for the context, ya'll. Since SIP-12 had not been voted on for adoption I was hoping to chime in before there was a lot of "sunk cost" and before it was locked. Your background was helpful, thanks.
I do want to note that the motivation for my comments was not to make running git commands and creating branches easy -- that is always easy. :-) It's about the flow of code and features through branches and how you manage those along with customer/support SLAs. The way you do that can make the things that ARE hard and potentially very time consuming -- validating, testing, fixing bugs, releasing, and supporting -- much, much more efficient. I'll take some time to absorb how folks are currently working day to day as I ramp up, then revisit later. I'll also take some time to dive into the Apache processes. I definitely should gain more understanding of that aspect. Thanks again! Dave On Tue, Mar 26, 2019 at 11:43 AM Krist Wongsuphasawat <krist.wo...@gmail.com> wrote: > Hi Dave, > > > We appreciate your comments on the release process and are glad to have > someone with experience about this topic on board. Before going deep into > the discussion about gitflow, we would like to clarify our stance on the > topic. > > > First of all, git workflow and cherry-pick are not the pain point in the > current release process. As you can verify with Christine and others who > have worked on the past releases that the git commands and creating > branches are easy parts of the release. The time-consuming parts are > verifying features, finding bugs and fixing issues. Therefore, changing the > git workflow is not the main item on the top of our wishlist at the moment. > If you would like to provide command line tools for git tagging, creating > change logs, that would be greatly welcomed. However, please keep in mind > that the git tasks probably accounts for <10% of the work that actually > happens in each release, and the 90% are the parts we hope can be more > efficient. > > > Secondly, we do not object if you would like to improve the git process. > However, we have invested a lot of effort into drafting SIP-12 and shifting > our dev environment to that workflow already. The SIP has been posted since > last year. We would like to save our bandwidth to focus on other topics > rather than changing the git process. If you would like to come up with a > new process, please also think of a smooth transition between the current > plan to that plan and what are the benefits we all would gain. > > > Third, making the release process compatible with Apache release protocols. > This is the part where we have not carefully thought about and would be a > great area where your expertise could be helpful. We do not have any > problem with having 18 RCs for internal releases and testing. But if having > frequent version bump like this is not a viable option for Apache > “vote-before-release” protocol, we need to come up with some solutions. > > > Thank you again for your initiation. We are happy to have you on board and > hope your experience can help all of us make the releases better. > > > Best regards, > > > MIchelle and Krist > > > > On Fri, Mar 22, 2019 at 10:32 AM David Smith <dave.a.sm...@gmail.com> > wrote: > > > Thanks, Michelle. The reliance on cherry-picking as part of a work flow > can > > definitely bring the process into conflict with other tools and processes > > in the git ecosystem. I've left a fairly lengthy comment on SIP-12 > > proposing that the mechanics be handled using gitflow, which does not use > > cherry-picking, just for discussion. > > > > > https://github.com/apache/incubator-superset/issues/6131#issuecomment-475691949 > > > > On Wed, Mar 20, 2019 at 1:04 PM Michelle Thomas > > <michelle.tho...@airbnb.com.invalid> wrote: > > > > > To answer the question about using a workflow for managing > > > releases/branches/tags, I previously evaluated using conventional > > commits: > > > https://www.conventionalcommits.org/en/v1.0.0-beta.3/, which is a > system > > > for tagging commits to generate the changelog. At the time it seemed > like > > > it may have issues creating the Changelog with the way we commit > > everything > > > to master then cherry-pick fixes onto a branch, but the consistency in > > > tagging PRs with fix, feature, docs seemed useful. I don't think we've > > > heavily discussed other workflows. Sounds good to get suggestions for > > > improvement on SIP-12. > > > > > > On Tue, Mar 19, 2019 at 9:27 PM David Smith <dave.a.sm...@gmail.com> > > > wrote: > > > > > > > I'm new to this project, so I apologize if this has been discussed > > > > previously, but... Have we considered using a widely adopted workflow > > for > > > > managing releases/branches/tags, and using tools that the community > has > > > > already built for them? For example Git Flow or some variant > thereof? > > > For > > > > anyone who is unfamiliar, see overview and tooling--in the form of > git > > > > extensions--here: https://github.com/nvie/gitflow > > > > > > > > I saw SIP-12 and have a few comments that I will try to document > > tomorrow > > > > on that issue, but I'm really curious whether other common > > > processes/flows > > > > were rejected explicitly or if they just didn't come up? > > > > > > > > Dave > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 8:45 PM Maxime Beauchemin < > > > > maximebeauche...@gmail.com> wrote: > > > > > > > > > Wondering what to do next, Justin, is it ok for me to push the > > related > > > > tag > > > > > to Github? > > > > > > > > > > Should I trigger a [VOTE] thread? > > > > > > > > > > Max > > > > > > > > > > On Tue, Mar 19, 2019 at 12:04 AM Maxime Beauchemin < > > > > > maximebeauche...@gmail.com> wrote: > > > > > > > > > > > My svn is a bit rusty but I made some progress tonight: > > > > > > https://github.com/apache/incubator-superset/pull/7054 > > > > > > > > > > > > Pushed some files to SVN > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/superset/0.31.0rc18/ > > > > > > > > > > > > It's not intended to be an actual real RC, but it's a start. > > > > > > > > > > > > An early question is around RC numbers. Lyft & Airbnb have been > > > doing a > > > > > > lot of back and forth on the release branch already and `0.31.0` > is > > > > > already > > > > > > up to rc18. I was internally debating what to do here, keeping > the > > > > > > numbering scheme, or starting anew. Probably the right thing to > do > > is > > > > to > > > > > > pick a number (say 0.32.0) and start clean at 0.32.0rc1, and > stick > > to > > > > the > > > > > > Apache process, and use continuous numbers. I'm assuming the > > handoff > > > is > > > > > at > > > > > > 0.31.0rc18. > > > > > > > > > > > > Now people that would want to work on non-Apache release branches > > > would > > > > > do > > > > > > so in their fork, and in their own namespace. Maybe their local > > build > > > > > > instructions can become recipes for an upcoming release, but I > > > suggest > > > > > that > > > > > > this happens in forks from now on. > > > > > > > > > > > > Clearly this Apache process here with signing + svn commit + > voting > > > is > > > > > > more involved than the usual git cherry-pick + git tag + git > push, > > so > > > > > that > > > > > > should probably lead to less RCs (I can't imagine doing 18 votes > > > > around a > > > > > > single release). > > > > > > > > > > > > Not to mention the fact that the real complexity around releasing > > is > > > > > > raising and labeling issues with a proper target version, gather > > PRs, > > > > > > cherry-pick fixes and resolve conflicts if any. I'm hoping we can > > > build > > > > > > tooling to help with all this. Hugh started something a while > back, > > > but > > > > > > there's lots to be done still in that area. > > > > > > > > > > > > Max > > > > > > > > > > > > On Mon, Mar 18, 2019 at 11:39 PM Maxime Beauchemin < > > > > > > maximebeauche...@gmail.com> wrote: > > > > > > > > > > > >> Not officially approved yet, I think we should start a [DISCUSS] > > > > thread > > > > > >> and eventually [VOTE] > > > > > >> > > > > > >> On Mon, Mar 18, 2019 at 10:13 PM David Smith < > > > dave.a.sm...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > >>> Thanks Max! Out of curiosity, has that release SIP-12 been > > approved > > > > > yet? > > > > > >>> I > > > > > >>> have some thoughts but if it is already a done deal I'll wait > > until > > > > > >>> another > > > > > >>> time. :-) > > > > > >>> > > > > > >>> On Mon, Mar 18, 2019 at 9:53 PM Maxime Beauchemin < > > > > > >>> maximebeauche...@gmail.com> wrote: > > > > > >>> > > > > > >>> > Hi all, > > > > > >>> > > > > > > >>> > I wanted to send an email explaining the current state of > > > releases > > > > > and > > > > > >>> > start a thread about what's ahead. > > > > > >>> > > > > > > >>> > Early on in the project lifecycle, I used to package releases > > and > > > > > push > > > > > >>> to > > > > > >>> > Pypi without much process. I'd package internal releases > > > internally > > > > > for > > > > > >>> > Airbnb. We'd roll out to staging, stabilize the release, and > > > launch > > > > > in > > > > > >>> > production. After a some time without major issues and > > > regressions, > > > > > I'd > > > > > >>> > push a new release out to Pypi and make it available to the > > > > > community. > > > > > >>> That > > > > > >>> > process worked ok when I reviewed or wrote most PRs and had a > > > > handle > > > > > on > > > > > >>> > everything that was going on. The community has grown much, > and > > > > > clearly > > > > > >>> > that process has not been appropriate for quite some time > now. > > > > > >>> > > > > > > >>> > Later on, we joined the Apache Software Foundation's > incubator, > > > and > > > > > >>> the ASF > > > > > >>> > has clear process around releases > > > > > >>> > <http://www.apache.org/dev/release-publishing.html>. As an > ASF > > > > > >>> project, it > > > > > >>> > was not reasonable for us to push to Pypi until we sorted out > > the > > > > > >>> process > > > > > >>> > and followed the "Apache Way". Since `0.29` we've still been > > > > cutting > > > > > >>> > release branches and labeling "release candidates", but have > > > > > refrained > > > > > >>> from > > > > > >>> > publishing to Pypi. Those release branches contain commits > that > > > > have > > > > > >>> been > > > > > >>> > put into production at Lyft and/or Airbnb and elsewhere, but > > > didn't > > > > > >>> meet > > > > > >>> > the ASF's standards, so could not be published as releases. > > > > > >>> > > > > > > >>> > Also with the growing community, we clearly need a more > > > structured > > > > > >>> process > > > > > >>> > around releases. John Bodley wrote a solid SIP > > > > > >>> > <https://github.com/apache/incubator-superset/issues/6131> > > > > > proposing a > > > > > >>> > clear structure and process around releases, and details much > > of > > > > what > > > > > >>> the > > > > > >>> > ASF release process does not. > > > > > >>> > > > > > > >>> > All this to say, *I'm starting some work on our first > > > ASF-approved > > > > > >>> source > > > > > >>> > code release*. For context, the ASF makes a distinction > between > > > > > "source > > > > > >>> > code releases" and "convenience releases". The former > contains > > > the > > > > > >>> source > > > > > >>> > code and build instructions, and the latter contains binaries > > and > > > > is > > > > > >>> made > > > > > >>> > available in places like Pypi (or Maven / npm / RubyGem / > ...). > > > > > >>> > > > > > > >>> > While the source code release should be pretty straight > > forward, > > > > it's > > > > > >>> not > > > > > >>> > really, well, convenient. You can't just "pip install" > that... > > > The > > > > > >>> > convenience release on the other hand is great, but may > require > > > > more > > > > > >>> work > > > > > >>> > and guidance from our mentors, ASF legal, license experts as > > the > > > > > >>> binaries > > > > > >>> > may contain our Javascript bundles (or maybe they don't?) but > > > then > > > > > >>> have to > > > > > >>> > have a gigantic LICENSE.txt file detailing the licenses of > the > > > 500+ > > > > > >>> > Javascript libs that have accumulated in our > > `package-lock.json` > > > > over > > > > > >>> time. > > > > > >>> > Or maybe we ship without the bundles and add a `superset > > compile` > > > > > >>> command > > > > > >>> > that does the fetching/build of the JS packages. Unclear to > me. > > > > I'll > > > > > >>> need > > > > > >>> > help figuring this out. For context here's some work I did > > trying > > > > to > > > > > >>> > automate the LICENSE.txt creation > > > > > >>> > <https://github.com/apache/incubator-superset/pull/5801> > > > > > >>> > > > > > > >>> > Anyhow, wanted people to know what's up since it's been so > long > > > > since > > > > > >>> the > > > > > >>> > last Pypi release, and give the opportunity for contributors > to > > > > jump > > > > > >>> in and > > > > > >>> > help out. > > > > > >>> > > > > > > >>> > Let's get the [community] release process going! > > > > > >>> > > > > > > >>> > Max > > > > > >>> > > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > -- > > *Krist Wongsuphasawat, PhD* > http://kristw.yellowpigz.com >
