I agree. In my experiences as RM I have found the involvement of Maven
in the release process to be a nuisance. I think it makes more sense
in Java-only projects

On Thu, Aug 1, 2019 at 2:54 PM Andy Grove <andygrov...@gmail.com> wrote:
>
> I'll start taking a look at the maven issue. We might not want to use maven
> release plugin given that we control the version number already across this
> repository via other means.
>
> On Wed, Jul 31, 2019 at 4:26 PM Sutou Kouhei <k...@clear-code.com> wrote:
>
> > Hi,
> >
> > Sorry for not replying this thread.
> >
> > I think that the biggest problem is related to our Java
> > package.
> >
> >
> > We'll be able to resolve the GPG key problem by creating a
> > GPG key only for nightly release test. We can share the test
> > GPG key publicly because it's a just for testing.
> >
> > It'll work for our binary artifacts and APT/Yum repositories
> > but not work for our Java package. I don't know where GPG
> > key is used in our Java package...
> >
> >
> > We'll be able to resolve the Git commit problem by creating
> > a cloned Git repository for test. It's done in our
> > dev/release/00-prepare-test.rb[1].
> >
> > [1]
> > https://github.com/apache/arrow/blob/master/dev/release/00-prepare-test.rb#L30
> >
> > The biggest problem for the Git commit is our Java package
> > requires "apache-arrow-${VERSION}" tag on
> > https://github.com/apache/arrow . (Right?)
> > I think that "mvm release:perform" in
> > dev/release/01-perform.sh does so but I don't know the
> > details of "mvm release:perform"...
> >
> >
> > More details:
> >
> > dev/release/00-prepare.sh:
> >
> > We'll be able to run this automatically when we can resolve
> > the above GPG key problem in our Java package. We can
> > resolve the Git commit problem by creating a cloned Git
> > repository.
> >
> > dev/release/01-prepare.sh:
> >
> > We'll be able to run this automatically when we can resolve
> > the above Git commit ("apche-arrow-${VERSION}" tag) problem
> > in our Java package.
> >
> > dev/release/02-source.sh:
> >
> > We'll be able to run this automatically by creating a GPG
> > key for nightly release test. We'll use Bintray to upload RC
> > source archive instead of dist.apache.org. Ah, we need a
> > Bintray API key for this. It must be secret.
> >
> > dev/release/03-binary.sh:
> >
> > We'll be able to run this automatically by creating a GPG
> > key for nightly release test. We need a Bintray API key too.
> >
> > We need to improve this to support nightly release test. It
> > use "XXX-rc" such as "debian-rc" for Bintray "package" name.
> > It should use "XXX-nightly" such as "debian-nightly" for
> > nightly release test instead.
> >
> > dev/release/post-00-release.sh:
> >
> > We'll be able to skip this.
> >
> > dev/release/post-01-upload.sh:
> >
> > We'll be able to skip this.
> >
> > dev/release/post-02-binary.sh:
> >
> > We'll be able to run this automatically by creating Bintray
> > "packages" for nightly release and use them. We can create
> > "XXX-nightly-release" ("debian-nightly-release") Bintray
> > "packages" and use them instead of "XXX" ("debian") Bintray
> > "packages".
> >
> > "debian" Bintray "package": https://bintray.com/apache/debian/
> >
> > We need to improve this to support nightly release.
> >
> > dev/release/post-03-website.sh:
> >
> > We'll be able to run this automatically by creating a cloned
> > Git repository for test.
> >
> > It's better that we have a Web site to show generated pages.
> > We can create
> > https://github.com/apache/arrow-site/tree/asf-site/nightly
> > and use it but I don't like it. Because arrow-site increases
> > a commit day by day.
> > Can we prepare a Web site for this? (arrow-nightly.ursalabs.org?)
> >
> > dev/release/post-04-rubygems.sh:
> >
> > We may be able to use GitHub Package Registry[2] to upload
> > RubyGems. We can use "pre-release" package feature of
> > https://rubygems.org/ but it's not suitable for
> > nightly. It's for RC or beta release.
> >
> > [2] https://github.blog/2019-05-10-introducing-github-package-registry/
> >
> > dev/release/post-05-js.sh:
> >
> > We may be able to use GitHub Package Registry[2] to upload
> > npm packages.
> >
> > dev/release/post-06-csharp.sh:
> >
> > We may be able to use GitHub Package Registry[2] to upload
> > NuGet packages.
> >
> > dev/release/post-07-rust.sh:
> >
> > I don't have any idea. But it must be ran
> > automatically. It's always failed. I needed to run each
> > command manually.
> >
> > dev/release/post-08-remove-rc.sh:
> >
> > We'll be able to skip this.
> >
> >
> > Thanks,
> > --
> > kou
> >
> > In <CAJPUwMAmp0jvz6qrdqehBEUB_NdbGEsHFNgLW9Q6V9RFTnk=3...@mail.gmail.com>
> >   "Re: [DISCUSS] Release cadence and release vote conventions" on Wed, 31
> > Jul 2019 15:35:57 -0500,
> >   Wes McKinney <wesmck...@gmail.com> wrote:
> >
> > > The PMC member and their GPG keys need to be in the loop at some
> > > point. The release artifacts can be produced by some kind of CI/CD
> > > system so long as the PMC member has confidence in the security of
> > > those artifacts before signing them. For example, we build the
> > > official binary packages on public CI services and then download and
> > > sign them with Crossbow. I think the same could be done in theory with
> > > the source release but we'd first need to figure out what to do about
> > > the parts that create git commits.
> > >
> > > On Wed, Jul 31, 2019 at 11:23 AM Andy Grove <andygrov...@gmail.com>
> > wrote:
> > >>
> > >> To what extent would it be possible to automate the release process via
> > >> CICD?
> > >>
> > >> On Wed, Jul 31, 2019 at 9:19 AM Wes McKinney <wesmck...@gmail.com>
> > wrote:
> > >>
> > >> > I think one thing that would help would be improving the
> > >> > reproducibility of the source release process. The RM has to have
> > >> > their machine configured in a particular way for it to work.
> > >> >
> > >> > Before anyone says "Docker" it isn't an easy solution because the
> > >> > release scripts need to be able to create git commits (created by the
> > >> > Maven release plugin) and sign artifacts using the RM's GPG keys.
> > >> >
> > >> > On Sat, Jul 27, 2019 at 10:04 PM Micah Kornfield <
> > emkornfi...@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > I just wanted to bump this thread.  Kou and KrisztiƔn as the last
> > two
> > >> > > release managers is there any specific infrastructure that you think
> > >> > might
> > >> > > have helped?
> > >> > >
> > >> > > Thanks,
> > >> > > Micah
> > >> > >
> > >> > > On Wed, Jul 17, 2019 at 11:29 PM Micah Kornfield <
> > emkornfi...@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > I'd can help as well, but not exactly sure where to start.  It
> > seems
> > >> > like
> > >> > > > there are already some JIRAs opened [1]
> > >> > > > for improving the release?  Could someone more familiar with the
> > >> > process
> > >> > > > pick out the highest priority ones? Do more need to be opened?
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Micah
> > >> > > >
> > >> > > > [1]
> > >> > > >
> > >> >
> > https://issues.apache.org/jira/browse/ARROW-2880?jql=project%20%3D%20ARROW%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20component%20in%20(%22Developer%20Tools%22%2C%20Packaging)%20and%20summary%20~%20Release
> > >> > > >
> > >> > > > On Sat, Jul 13, 2019 at 7:17 AM Wes McKinney <wesmck...@gmail.com
> > >
> > >> > wrote:
> > >> > > >
> > >> > > >> To be effective at improving the life of release managers, the
> > nightly
> > >> > > >> release process really should use as close as possible to the
> > same
> > >> > > >> scripts that the RM uses to produce the release. Otherwise we
> > could
> > >> > > >> have a situation where the nightlies succeed but there is some
> > problem
> > >> > > >> that either fails an RC or is unable to be produced at all.
> > >> > > >>
> > >> > > >> On Sat, Jul 13, 2019 at 9:12 AM Andy Grove <
> > andygrov...@gmail.com>
> > >> > wrote:
> > >> > > >> >
> > >> > > >> > I would like to volunteer to help with Java and Rust release
> > process
> > >> > > >> work,
> > >> > > >> > especially nightly releases.
> > >> > > >> >
> > >> > > >> > Although I'm not that familiar with the Java implementation of
> > >> > Arrow, I
> > >> > > >> > have been using Java and Maven for a very long time.
> > >> > > >> >
> > >> > > >> > Do we envisage a single nightly release process that releases
> > all
> > >> > > >> languages
> > >> > > >> > simultaneously? or do we want separate process per language,
> > with
> > >> > > >> different
> > >> > > >> > maintainers?
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > On Wed, Jul 10, 2019 at 8:18 AM Wes McKinney <
> > wesmck...@gmail.com>
> > >> > > >> wrote:
> > >> > > >> >
> > >> > > >> > > On Sun, Jul 7, 2019 at 7:40 PM Sutou Kouhei <
> > k...@clear-code.com>
> > >> > > >> wrote:
> > >> > > >> > > >
> > >> > > >> > > > Hi,
> > >> > > >> > > >
> > >> > > >> > > > > in future releases we should
> > >> > > >> > > > > institute a minimum 24-hour "quiet period" after any
> > community
> > >> > > >> > > > > feedback on a release candidate to allow issues to be
> > examined
> > >> > > >> > > > > further.
> > >> > > >> > > >
> > >> > > >> > > > I agree with this. I'll do so when I do a release manager
> > in
> > >> > > >> > > > the future.
> > >> > > >> > > >
> > >> > > >> > > > > To be able to release more often, two things have to
> > happen:
> > >> > > >> > > > >
> > >> > > >> > > > > * More PMC members must engage with the release
> > management
> > >> > role,
> > >> > > >> > > > > process, and tools
> > >> > > >> > > > > * Continued improvements to release tooling to make the
> > >> > process
> > >> > > >> less
> > >> > > >> > > > > painful for the release manager. For example, it seems
> > we may
> > >> > > >> want to
> > >> > > >> > > > > find a different place than Bintray to host binary
> > artifacts
> > >> > > >> > > > > temporarily during release votes
> > >> > > >> > > >
> > >> > > >> > > > My opinion that we need to build nightly release system.
> > >> > > >> > > >
> > >> > > >> > > > It uses dev/release/NN-*.sh to build .tar.gz and binary
> > >> > > >> > > > artifacts from the .tar.gz.
> > >> > > >> > > > It also uses dev/release/verify-release-candidate.* to
> > >> > > >> > > > verify build .tar.gz and binary artifacts.
> > >> > > >> > > > It also uses dev/release/post-NN-*.sh to do post release
> > >> > > >> > > > tasks. (Some tasks such as uploading a package to packaging
> > >> > > >> > > > system will be dry-run.)
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> > > I agree that having a turn-key release system that's capable
> > of
> > >> > > >> > > producing nightly packages is the way to do. That way any
> > problems
> > >> > > >> > > that would block a release will come up as they happen
> > rather than
> > >> > > >> > > piling up until the very end like they are now.
> > >> > > >> > >
> > >> > > >> > > > I needed 10 or more changes for dev/release/ to create
> > >> > > >> > > > 0.14.0 RC0. (Some of them are still in my local stashes. I
> > >> > > >> > > > don't have time to create pull requests for them
> > >> > > >> > > > yet. Because I postponed some tasks of my main
> > >> > > >> > > > business. I'll create pull requests after I finished the
> > >> > > >> > > > postponed tasks of my main business.)
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> > > Thanks. I'll follow up on the 0.14.1/0.15.0 thread -- since
> > we
> > >> > need to
> > >> > > >> > > release again soon because of problems with 0.14.0 please
> > let us
> > >> > know
> > >> > > >> > > what patches will be needed to make another release.
> > >> > > >> > >
> > >> > > >> > > > If we fix problems related to dev/release/ in our normal
> > >> > > >> > > > development process, release process will be less painful.
> > >> > > >> > > >
> > >> > > >> > > > The biggest problem for 0.14.0 RC0 is java/pom.xml related:
> > >> > > >> > > >   https://github.com/apache/arrow/pull/4717
> > >> > > >> > > >
> > >> > > >> > > > It was difficult for me because I don't have Java
> > >> > > >> > > > knowledge. Release manager needs help from many developers
> > >> > > >> > > > because release manager may not have knowledge of all
> > >> > > >> > > > supported languages. Apache Arrow supports 10 over
> > >> > > >> > > > languages.
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > > For Bintray API limit problem, we'll be able to resolve it.
> > >> > > >> > > > I was added to https://bintray.com/apache/ members:
> > >> > > >> > > >
> > >> > > >> > > >   https://issues.apache.org/jira/browse/INFRA-18698
> > >> > > >> > > >
> > >> > > >> > > > I'll be able to use Bintray API without limitation in the
> > >> > > >> > > > future. Release managers should also request the same
> > thing.
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> > > This is good, I will add myself. Other PMC members should
> > also add
> > >> > > >> > > themselves.
> > >> > > >> > >
> > >> > > >> > > >
> > >> > > >> > > > Thanks,
> > >> > > >> > > > --
> > >> > > >> > > > kou
> > >> > > >> > > >
> > >> > > >> > > > In <CAJPUwMBRzYQ=hbVwFuPYAB-O=
> > >> > > >> lsowxqxidjapc_cofguksj...@mail.gmail.com>
> > >> > > >> > > >   "[DISCUSS] Release cadence and release vote conventions"
> > on
> > >> > Sat,
> > >> > > >> 6 Jul
> > >> > > >> > > 2019 16:28:50 -0500,
> > >> > > >> > > >   Wes McKinney <wesmck...@gmail.com> wrote:
> > >> > > >> > > >
> > >> > > >> > > > > hi folks,
> > >> > > >> > > > >
> > >> > > >> > > > > As a reminder, particularly since we have many new
> > community
> > >> > > >> members
> > >> > > >> > > > > (some of whom have never been involved with an ASF
> > project
> > >> > > >> before),
> > >> > > >> > > > > releases are approved exclusively by the PMC and in
> > general
> > >> > > >> releases
> > >> > > >> > > > > cannot be vetoed. In spite of that, we strive to make
> > releases
> > >> > > >> that
> > >> > > >> > > > > have unanimous (either by explicit +1 or lazy consent)
> > >> > support of
> > >> > > >> the
> > >> > > >> > > > > PMC. So it is better to have unanimous 5 +1 votes than 6
> > +1
> > >> > votes
> > >> > > >> with
> > >> > > >> > > > > a -1 dissenting vote.
> > >> > > >> > > > >
> > >> > > >> > > > > On the 0.14.0 vote, as with previous release votes, some
> > >> > issues
> > >> > > >> with
> > >> > > >> > > > > the release were raised by members of the community,
> > whether
> > >> > > >> build or
> > >> > > >> > > > > test-related problems or other failures. Technically
> > speaking,
> > >> > > >> such
> > >> > > >> > > > > issues have no _direct_ bearing on whether a release vote
> > >> > passes,
> > >> > > >> only
> > >> > > >> > > > > on whether PMC members vote +1, 0, or -1. A PMC member is
> > >> > allowed
> > >> > > >> to
> > >> > > >> > > > > change their vote based on new information -- for
> > example, if
> > >> > I
> > >> > > >> voted
> > >> > > >> > > > > +1 on a release and then someone reported a serious
> > licensing
> > >> > > >> issue,
> > >> > > >> > > > > then I would revise my vote to -1.
> > >> > > >> > > > >
> > >> > > >> > > > > On the RC0 vote thread, Jacques wrote [1]
> > >> > > >> > > > >
> > >> > > >> > > > > "A release vote should last until we arrive at consensus.
> > >> > When an
> > >> > > >> > > > > issue is potentially identified, those that have voted
> > should
> > >> > be
> > >> > > >> given
> > >> > > >> > > > > ample time to change their vote and others that may have
> > been
> > >> > lazy
> > >> > > >> > > > > consenters should be given time to chime in. There is no
> > >> > maximum
> > >> > > >> > > > > amount of time a vote can be open. Allowing at least 24
> > hours
> > >> > > >> after an
> > >> > > >> > > > > objection is raised is a pretty minimum expectation
> > unless the
> > >> > > >> > > > > objector removes their objection.
> > >> > > >> > > > >
> > >> > > >> > > > > Note that Apache is more focused on consensus than
> > timing (as
> > >> > > >> opposed
> > >> > > >> > > to
> > >> > > >> > > > > virtually other other organizations in the world)."
> > >> > > >> > > > >
> > >> > > >> > > > > I agree with this and my opinion is that in future
> > releases we
> > >> > > >> should
> > >> > > >> > > > > institute a minimum 24-hour "quiet period" after any
> > community
> > >> > > >> > > > > feedback on a release candidate to allow issues to be
> > examined
> > >> > > >> > > > > further. If someone finds a potential problem, and no
> > negative
> > >> > > >> votes
> > >> > > >> > > > > are cast or changed, then the vote can close.
> > >> > > >> > > > >
> > >> > > >> > > > > As a related matter, it seems clear to me that Apache
> > Arrow
> > >> > should
> > >> > > >> > > > > have more frequent releases. I think this would decrease
> > >> > pressure
> > >> > > >> on
> > >> > > >> > > > > developers and users alike. While we've made strides to
> > >> > improve
> > >> > > >> the
> > >> > > >> > > > > tooling for release management (big thanks to Kou,
> > Yosuke,
> > >> > > >> Krisztian,
> > >> > > >> > > > > and others), there is still quite some labor involved and
> > >> > > >> potential
> > >> > > >> > > > > for issues (e.g. API rate limiting for binary artifacts
> > on
> > >> > > >> Bintray).
> > >> > > >> > > > >
> > >> > > >> > > > > To be able to release more often, two things have to
> > happen:
> > >> > > >> > > > >
> > >> > > >> > > > > * More PMC members must engage with the release
> > management
> > >> > role,
> > >> > > >> > > > > process, and tools
> > >> > > >> > > > > * Continued improvements to release tooling to make the
> > >> > process
> > >> > > >> less
> > >> > > >> > > > > painful for the release manager. For example, it seems
> > we may
> > >> > > >> want to
> > >> > > >> > > > > find a different place than Bintray to host binary
> > artifacts
> > >> > > >> > > > > temporarily during release votes
> > >> > > >> > > > >
> > >> > > >> > > > > Any other ideas for things we can do to improve the
> > process
> > >> > and
> > >> > > >> > > > > cadence of releases?
> > >> > > >> > > > >
> > >> > > >> > > > > Thanks,
> > >> > > >> > > > > Wes
> > >> > > >> > > > >
> > >> > > >> > > > > [1]:
> > >> > > >> > >
> > >> > > >>
> > >> >
> > https://lists.apache.org/thread.html/be6210e97b838494a5516dad6408f479efe4c98aff805000597c0196@%3Cdev.arrow.apache.org%3E
> > >> > > >> > >
> > >> > > >>
> > >> > > >
> > >> >
> >

Reply via email to