Hi,
No objection. I'll start a vote for this.
Thanks,
--
kou
In <20240820.091745.710358189219212077....@clear-code.com>
"Re: [DISCUSS] Split Go release process" on Tue, 20 Aug 2024 09:17:45 +0900
(JST),
Sutou Kouhei <k...@clear-code.com> wrote:
> Hi,
>
>> I think we'll need a vote on it *eventually*, but we probably can wait
>> until the repo is up and ready with the vote being when we "pull the
>> trigger" so-to-say and turn off Go releases from the main
>> github.com/apache/arrow repository and start releasing through
>> github.com/apache/arrow-go. We'll also probably need to shift open PRs /
>> issues to the new repo after we do this.
>
> OK. I'll start a vote next week if nobody objects this
> proposal in this week. I think that a vote after preparing
> github.com/apache/arrow-go may inhibit -1 vote. So a vote
> before preparing github.com/apache/arrow-go will be fair.
>
>
> Thanks,
> --
> kou
>
> In <cah4123bccmr2lgvprbsjc7-sktsfy5bwomeseafosrbwyph...@mail.gmail.com>
> "Re: [DISCUSS] Split Go release process" on Mon, 19 Aug 2024 16:15:52 -0400,
> Matt Topol <zotthewiz...@gmail.com> wrote:
>
>>> Based on 6., users need to change their import paths on
>>> upgrade whether we keep using apache/arrow or we use new
>>> apache/arrow-go.
>>>
>>> If we use new apache/arrow-go, we will be able to reduce
>>> maintenance cost for apache/arrow (e.g. we can remove Go
>>> related scripts, CI jobs and so on from apache/arrow). Let's
>>> use apache/arrow-go.
>>>
>>> If nobody objects splitting Apache Arrow Go to
>>> apache/arrow-go in this week, I'll start working on this
>>> next week. (Do we need a vote for this?)
>>
>> I think we'll need a vote on it *eventually*, but we probably can wait
>> until the repo is up and ready with the vote being when we "pull the
>> trigger" so-to-say and turn off Go releases from the main
>> github.com/apache/arrow repository and start releasing through
>> github.com/apache/arrow-go. We'll also probably need to shift open PRs /
>> issues to the new repo after we do this.
>>
>> I'm happy to review any of the scripts being added to the new repository
>> and help out if you need it in getting this going. I think the prospect of
>> fewer major releases, and thus easier upgrades, is very worthwhile for us
>> to pursue this.
>>
>> --Matt
>>
>> On Mon, Aug 19, 2024 at 1:14 AM Sutou Kouhei <k...@clear-code.com> wrote:
>>
>>> Hi,
>>>
>>> Sorry for not working on this.
>>>
>>> Thanks for sharing the standard docs! I've read it and
>>> related docs.
>>>
>>> Here is the summary I learned in this thread and the
>>> standard docs:
>>>
>>> 1. We're using "github.com/apache/arrow/go/v${VERSION}
>>> <http://github.com/apache/arrow/go/v$%7BVERSION%7D>" such
>>> as "github.com/apache/arrow/go/v17" as our module name
>>> * https://pkg.go.dev/github.com/apache/arrow/go/v17/arrow
>>> * Including the version number part ("v${VERSION}") is
>>> important
>>> * Users can avoid unexpected backward incompatibility by
>>> this style
>>> 2. We used to use "github.com/apache/arrow/go" as our module
>>> name in v5 or earlier
>>> * https://pkg.go.dev/github.com/apache/arrow/go/arrow
>>> * 133 modules still use this
>>> 3. We want to avoid user side changes as much as possible
>>> * As 2. shows, users may keep using old version if there
>>> is any change is required
>>> 4. The current users need to change Apache Arrow Go's import
>>> path to "github.com/apache/arrow/go/v${VERSION
>>> <http://github.com/apache/arrow/go/v$%7BVERSION> + 1}" when
>>> they want to upgrade Apache Arrow Go
>>> * We don't want to require more changes than "changing
>>> import path" for users as mentioned in 3.
>>> 5. We can't provide backward compatible module name such as
>>> "github.com/apache/arrow/go/v18" for
>>> "github.com/apache/arrow-go/v18"
>>> * Go doesn't provide the feature
>>> 6. We want to keep "v${VERSION}" in our module name even if
>>> we split Apache Arrow Go to apache/arrow-go
>>> * It's for avoiding unexpected backward incompatibility
>>> in users' projects
>>>
>>>
>>> Based on 6., users need to change their import paths on
>>> upgrade whether we keep using apache/arrow or we use new
>>> apache/arrow-go.
>>>
>>> If we use new apache/arrow-go, we will be able to reduce
>>> maintenance cost for apache/arrow (e.g. we can remove Go
>>> related scripts, CI jobs and so on from apache/arrow). Let's
>>> use apache/arrow-go.
>>>
>>> If nobody objects splitting Apache Arrow Go to
>>> apache/arrow-go in this week, I'll start working on this
>>> next week. (Do we need a vote for this?)
>>>
>>>
>>> Thanks,
>>> --
>>> kou
>>>
>>> In <cah4123zxadcug6yrkz2mxupke1muftyrvhg0hh1bqck5fw+...@mail.gmail.com>
>>> "Re: [DISCUSS] Split Go release process" on Mon, 22 Jul 2024 20:47:57
>>> -0400,
>>> Matt Topol <zotthewiz...@gmail.com> wrote:
>>>
>>> > Hey Kou,
>>> >
>>> > https://go.dev/doc/modules/release-workflow is the standard docs for
>>> > developing module versioning and publishing with Go.
>>> >
>>> > There isn't really a way to alias an import path to a different git repo
>>> > because it uses the GitHub URL itself as the import path.
>>> >
>>> > But it does seem like people seem to prefer the idea of shifting the Go
>>> > implementation to its own repository. I'd still push for us to include
>>> the
>>> > major version number in the import path, and since we'll have fewer major
>>> > releases and more minor releases, users shouldn't have to update their
>>> > import paths as frequently.
>>> >
>>> > --Matt
>>> >
>>> > On Mon, Jul 22, 2024, 8:37 PM Sutou Kouhei <k...@clear-code.com> wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> > Kou, is your plan also counting on moving the
>>> >> > specific nightlies there and removing them from the main repo?
>>> >>
>>> >> Yes. I should have mentioned it explicitly.
>>> >>
>>> >> We will remove most Go related CI jobs from apache/arrow. We
>>> >> will keep Go in integration test CI jobs like we do for
>>> >> apache/arrow-rs.
>>> >>
>>> >>
>>> >> Thanks,
>>> >> --
>>> >> kou
>>> >>
>>> >> In <cad1rbrr2vtxaunppfrrjgfd+ofca3q4f+yr6npku4ttzlx2...@mail.gmail.com>
>>> >> "Re: [DISCUSS] Split Go release process" on Fri, 19 Jul 2024 17:14:25
>>> >> +0200,
>>> >> Raúl Cumplido <raulcumpl...@gmail.com> wrote:
>>> >>
>>> >> > Hi,
>>> >> >
>>> >> > The conversation around more frequent minor releases and version split
>>> >> > per component has been a long one.
>>> >> >
>>> >> > I am in favour of these changes for the Go implementation because we
>>> >> > have several maintainers.
>>> >> >
>>> >> > It might be difficult to release other implementations that do not
>>> >> > have the same amount of maintainers. I am not sure what our plan is if
>>> >> > one of the split implementations has less maintainers and there's a
>>> >> > requirement for a release (i.e. security fix) but that might be
>>> >> > something to consider in the future.
>>> >> >
>>> >> >> I would defer to Raul and Jacob to corroborate this, but because
>>> >> >> changes to the CI configuration and release verification scripts
>>> don't
>>> >> >> affect other implementations, I have been able to maintain that
>>> >> >> infrastructure myself without too much effort and don't have to lean
>>> >> >> on them for anything except reviews.
>>> >> >
>>> >> > I think releasing and maintaining release scripts / verifications per
>>> >> > component is much easier than for the mono repo. We currently have
>>> >> > over 200 nightly CI jobs in the mono repo that are required to pass
>>> >> > before releasing. Moving some of those to its own repo helps
>>> >> > maintainability. Kou, is your plan also counting on moving the
>>> >> > specific nightlies there and removing them from the main repo?
>>> >> >
>>> >> > I would be in favour of doing a new major release (v18) once the repo
>>> >> > and the changes are in-place to update the import path to something
>>> >> > like:
>>> >> > github.com/apache/arrow-go/v18
>>> >> >
>>> >> > This would avoid confusion with previous releases. We can then follow
>>> >> > up with patch/minor/major as required.
>>> >> >
>>> >> > I am also happy to help with the releases and infrastructure if
>>> >> > necessary as I've done with the main Arrow one (I can also help on
>>> >> > nanoarrow, adbc if necessary).
>>> >> >
>>> >> > Kind regards,
>>> >> > Raul
>>> >> >
>>> >> >
>>> >> >
>>> >> >>
>>> >> >> [1] https://github.com/apache/arrow-nanoarrow/pull/557
>>> >> >>
>>> >> >> On Thu, Jul 18, 2024 at 7:53 PM Matt Topol <zotthewiz...@gmail.com>
>>> >> wrote:
>>> >> >> >
>>> >> >> > Part of the goal of splitting out the release processes is that
>>> we'd
>>> >> be
>>> >> >> > able to do minor version releases more frequently instead of major
>>> >> version
>>> >> >> > releases.
>>> >> >> >
>>> >> >> > The general convention in the Go community is to include a major
>>> >> version
>>> >> >> > "v#" in the import path for all major versions past v1 so that if
>>> >> there's a
>>> >> >> > breaking change, it's explicit and prevents potential issues from
>>> >> different
>>> >> >> > major versions being used simultaneously. Being able to do minor
>>> >> version
>>> >> >> > releases more frequently would lead to not having to change the
>>> import
>>> >> >> > paths every 3-6 months, but only if we actually do a breaking
>>> change.
>>> >> >> >
>>> >> >> > On Thu, Jul 18, 2024, 3:55 PM George Godik <ggo...@gmail.com>
>>> wrote:
>>> >> >> >
>>> >> >> > > > If we shift the Go lib to a new/different import
>>> >> >> > > path we'll end up with the same problem where people will rely on
>>> >> older
>>> >> >> > > versions and an incorrect path.
>>> >> >> > >
>>> >> >> > > Major version upgrades already require changing the import paths
>>> by
>>> >> >> > > increasing the version. The proposed change would require
>>> everyone
>>> >> to go
>>> >> >> > > through a similar process one last time.
>>> >> >> > >
>>> >> >> > > > More to the point, there would be the question of whether or
>>> not
>>> >> we
>>> >> >> > > should port over the same major version
>>> >> >> > > number, i.e. `github.com/apache/arrow-go/v17`
>>> <http://github.com/apache/arrow-go/v17>
>>> >> <http://github.com/apache/arrow-go/v17>
>>> >> >> > > <http://github.com/apache/arrow-go/v17>
>>> >> >> > > <http://github.com/apache/arrow-go/v17> or something to that
>>> end?
>>> >> Or
>>> >> >> > > do we restart back at v1 (which I think would be confusing)?
>>> >> >> > >
>>> >> >> > > My vote - for whatever it's worth - would be to do away with the
>>> >> >> > > version-in-path naming convention and relying on the go
>>> >> version/package
>>> >> >> > > system for major upgrades.
>>> >> >> > >
>>> >> >> > > Benefits: I don't have to change import paths every 3-6months
>>> >> >> > >
>>> >> >> > > On Thu, Jul 18, 2024 at 3:34 PM Matt Topol <
>>> zotthewiz...@gmail.com>
>>> >> wrote:
>>> >> >> > >
>>> >> >> > > > My thoughts:
>>> >> >> > > >
>>> >> >> > > > > * Go doesn't depend on other components such as C++
>>> >> >> > > > > * Go has some active PMC member (Matt) and committer (Joel)
>>> >> >> > > > > * Could you become a release manager for Go?
>>> >> >> > > >
>>> >> >> > > > I'd happily be the release manager for the Go implementation.
>>> >> >> > > >
>>> >> >> > > > > Here is my idea how to proceed this:
>>> >> >> > > > >
>>> >> >> > > > > 1. Extract go/ in apache/arrow to apache/arrow-go like
>>> >> >> > > > > apache/arrow-rs
>>> >> >> > > > > * Filter go/ related commits from apache/arrow and create
>>> >> >> > > > > apache/arrow-go with them like we did for
>>> apache/arrow-rs
>>> >> >> > > > > * Remove go/ related codes from apache/arrow
>>> >> >> > > > > 2. Prepare integration test CI like apache/arrow-rs does:
>>> >> >> > > > >
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >>
>>> https://github.com/apache/arrow-rs/blob/master/.github/workflows/integration.yml
>>> >> >> > > > > 3. Prepare release script based on apache/arrow-julia,
>>> >> >> > > > > apache/arrow-adbc and/or
>>> apache/arrow-flight-sql-postgresql
>>> >> >> > > >
>>> >> >> > > > Personally I would prefer that we do not extract it to its own
>>> >> separate
>>> >> >> > > > repository purely because I don't want to change the import
>>> path
>>> >> for
>>> >> >> > > users
>>> >> >> > > > again. We already have this issue from before we introduced the
>>> >> major
>>> >> >> > > > version into the import path and shifted it up to allow for the
>>> >> Parquet
>>> >> >> > > lib
>>> >> >> > > > in the same repository. If you look at [1] you see that there's
>>> >> still
>>> >> >> > > over
>>> >> >> > > > 100 projects that never upgraded to v6 or higher because they
>>> are
>>> >> still
>>> >> >> > > > using the old import path. If we shift the Go lib to a
>>> >> new/different
>>> >> >> > > import
>>> >> >> > > > path we'll end up with the same problem where people will rely
>>> on
>>> >> older
>>> >> >> > > > versions and an incorrect path.
>>> >> >> > > >
>>> >> >> > > > If we as a community decide that splitting out the
>>> >> implementations all
>>> >> >> > > into
>>> >> >> > > > separate repositories is the best way forward, I won't hold it
>>> up
>>> >> by
>>> >> >> > > > strictly hammering on this. I'm just concerned about the
>>> >> realities and
>>> >> >> > > > difficulties of communicating the import path change, ensuring
>>> we
>>> >> don't
>>> >> >> > > > break any consumers, and ensuring that users still end up being
>>> >> able to
>>> >> >> > > > upgrade easily.
>>> >> >> > > >
>>> >> >> > > > > The import path could be "github.com/apache/arrow-go"
>>> instead
>>> >> of "
>>> >> >> > > > github.com/apache/arrow-go/arrow". Since go will allow users
>>> to
>>> >> use
>>> >> >> > > > `arrow.Abc` directly if user imports `
>>> github.com/apache/arrow-go` <http://github.com/apache/arrow-go>
>>> >> <http://github.com/apache/arrow-go>
>>> >> >> > > <http://github.com/apache/arrow-go>
>>> >> >> > > > <http://github.com/apache/arrow-go>
>>> >> >> > > > <http://github.com/apache/arrow-go>.
>>> >> >> > > >
>>> >> >> > > > The import path would still have to be `
>>> >> >> > > github.com/apache/arrow-go/arrow`
>>> <http://github.com/apache/arrow-go/arrow>
>>> >> <http://github.com/apache/arrow-go/arrow>
>>> >> >> > > <http://github.com/apache/arrow-go/arrow>
>>> >> >> > > > <http://github.com/apache/arrow-go/arrow>
>>> >> >> > > > since it would also contain the parquet implementation in `
>>> >> >> > > > github.com/apache/arrow-go/parquet`
>>> <http://github.com/apache/arrow-go/parquet>
>>> >> <http://github.com/apache/arrow-go/parquet>
>>> >> >> > > <http://github.com/apache/arrow-go/parquet>
>>> >> >> > > > <http://github.com/apache/arrow-go/parquet>. More to the
>>> point,
>>> >> there
>>> >> >> > > > would be the
>>> >> >> > > > question of whether or not we should port over the same major
>>> >> version
>>> >> >> > > > number, i.e. `github.com/apache/arrow-go/v17`
>>> <http://github.com/apache/arrow-go/v17>
>>> >> <http://github.com/apache/arrow-go/v17>
>>> >> >> > > <http://github.com/apache/arrow-go/v17>
>>> >> >> > > > <http://github.com/apache/arrow-go/v17> or something to that
>>> >> end? Or
>>> >> >> > > > do we restart back at v1 (which I think would be confusing)?
>>> >> >> > > >
>>> >> >> > > > --Matt
>>> >> >> > > >
>>> >> >> > > > [1]: https://pkg.go.dev/github.com/apache/arrow/go/arrow
>>> >> >> > > >
>>> >> >> > > > On Thu, Jul 18, 2024 at 7:33 AM Antoine Pitrou <
>>> >> anto...@python.org>
>>> >> >> > > wrote:
>>> >> >> > > >
>>> >> >> > > > >
>>> >> >> > > > > Hi Kou,
>>> >> >> > > > >
>>> >> >> > > > > Le 18/07/2024 à 11:33, Sutou Kouhei a écrit :
>>> >> >> > > > > >
>>> >> >> > > > > > Here is my idea how to proceed this:
>>> >> >> > > > > >
>>> >> >> > > > > > 1. Extract go/ in apache/arrow to apache/arrow-go like
>>> >> >> > > > > > apache/arrow-rs
>>> >> >> > > > > > * Filter go/ related commits from apache/arrow and
>>> create
>>> >> >> > > > > > apache/arrow-go with them like we did for
>>> >> apache/arrow-rs
>>> >> >> > > > > > * Remove go/ related codes from apache/arrow
>>> >> >> > > > > > 2. Prepare integration test CI like apache/arrow-rs does:
>>> >> >> > > > > >
>>> >> >> > > > >
>>> >> >> > > >
>>> >> >> > >
>>> >>
>>> https://github.com/apache/arrow-rs/blob/master/.github/workflows/integration.yml
>>> >> >> > > > > > 3. Prepare release script based on apache/arrow-julia,
>>> >> >> > > > > > apache/arrow-adbc and/or
>>> >> apache/arrow-flight-sql-postgresql
>>> >> >> > > > >
>>> >> >> > > > > I think this is a good idea, but I'm not part of the Go
>>> >> maintainers.
>>> >> >> > > > >
>>> >> >> > > > > > Cons of this idea:
>>> >> >> > > > > >
>>> >> >> > > > > > * This is a backward incompatible change
>>> >> >> > > > > > * Users need to change their "import" to
>>> >> >> > > > > > "github.com/apache/arrow-go/arrow" from
>>> >> >> > > > > > "github.com/apache/arrow/go/arrow"
>>> >> >> > > > >
>>> >> >> > > > > Is there no way to leave some kind of alias or redirection in
>>> >> the
>>> >> >> > > > > apache/arrow repository?
>>> >> >> > > > >
>>> >> >> > > > > Regards
>>> >> >> > > > >
>>> >> >> > > > > Antoine.
>>> >> >> > > > >
>>> >> >> > > >
>>> >> >> > >
>>> >>
>>>
