Re: [DISCUSS] Versioning and releases for apache/arrow components
Another possibility I'd like to float is doing this in ADBC first? My primary motivation is (1) from Joris's list: I'd like to bump a few components (Snowflake, maybe SQLite) to a "stable" version while leaving the others behind, and in this context I think it'd be much more helpful to users to differentiate between stable/experimental components. Admittedly this would not quite address the questions around versioning docs and so on. On Tue, Apr 9, 2024, at 22:04, Antoine Pitrou wrote: > It seems that perhaps this discussion should be rebooted for each > individual component, one at a time? > > Let's start with something simple and obvious, with some frequent > contribution activity, such as perhaps Go? > > > > Le 09/04/2024 à 14:27, Joris Van den Bossche a écrit : >> I am also in favor of this idea in general and in the principle, but >> (somewhat repeating others) I think we should be aware that this will >> create _more_ work overall for releasing (refactoring release scripts >> (at least initially), deciding which version to use for which >> component, etc), and not less, given that easing the burden of the >> release managers was mentioned as a goal. >> So we if pursue this, it should be for other benefits that we think this has: >> 1) because separate versions would be beneficial for users? (have a >> clearer messaging in the version number (eg no major new version if >> there were hardly any changes in a certain component, or use a >> versioning scheme more common in a certain language's ecosystem, ..?) >> 2) because it would actually allow separate releases, even though when >> initially always releasing in batch (eg being able to just do a bug >> fix for go, without having to also add a tag for all others) >> 3) because it would make the release process more manageable / easier >> to delegate? (and thus potentially easing the burden for an >> _individual_ release manager, although requiring more work overall) >> 4) .. other things I am missing? >> >>> We could simply release C++, R, Python and C/GLib together. >>> ... I think that versioning will require additional thinking for libraries like PyArrow >>> I think we should maybe focus on a few more obvious cases. [i.e. not C++ >>> and Python] >> >> Yes, agreed to not focus on those. At least for PyArrow, speaking as >> one of its maintainers, I am personally (at this moment) not really >> interested in dealing with the complexity of allowing a decoupled >> Arrow C++ and PyArrow build. >> >> Related to the docs: >> >>> There is a dedicated documentation page for this... though the >>> versioning of the docs themselves would become ill-defined: >>> https://arrow.apache.org/docs/status.html >> ... >>> I think it would be best to hold off on Java also, in part because >>> of how the Java docs are integrated with the C++ and Python docs and >>> controlled by the version selector menu. >> >> We should indeed consider how to handle the current documentation >> site. Previously, we actually did some work to split the sphinx docs >> (used for the format, dev docs, and for the Python/C++/(part of the) >> Java docs) into multiple sphinx projects that could be built >> separately (https://github.com/apache/arrow/issues/30627, >> https://github.com/apache/arrow/pull/11980), but we abandoned that >> work last year because not seeming worthwhile. But we could certainly >> revive that idea, for example to at least split the format docs (and >> let that have its own versioning based on the Format Version >> (currently 1.4)? or just only host a single, latest version?) >> >> Joris
Re: [DISCUSS] Versioning and releases for apache/arrow components
It seems that perhaps this discussion should be rebooted for each individual component, one at a time? Let's start with something simple and obvious, with some frequent contribution activity, such as perhaps Go? Le 09/04/2024 à 14:27, Joris Van den Bossche a écrit : I am also in favor of this idea in general and in the principle, but (somewhat repeating others) I think we should be aware that this will create _more_ work overall for releasing (refactoring release scripts (at least initially), deciding which version to use for which component, etc), and not less, given that easing the burden of the release managers was mentioned as a goal. So we if pursue this, it should be for other benefits that we think this has: 1) because separate versions would be beneficial for users? (have a clearer messaging in the version number (eg no major new version if there were hardly any changes in a certain component, or use a versioning scheme more common in a certain language's ecosystem, ..?) 2) because it would actually allow separate releases, even though when initially always releasing in batch (eg being able to just do a bug fix for go, without having to also add a tag for all others) 3) because it would make the release process more manageable / easier to delegate? (and thus potentially easing the burden for an _individual_ release manager, although requiring more work overall) 4) .. other things I am missing? We could simply release C++, R, Python and C/GLib together. ... I think that versioning will require additional thinking for libraries like PyArrow I think we should maybe focus on a few more obvious cases. [i.e. not C++ and Python] Yes, agreed to not focus on those. At least for PyArrow, speaking as one of its maintainers, I am personally (at this moment) not really interested in dealing with the complexity of allowing a decoupled Arrow C++ and PyArrow build. Related to the docs: There is a dedicated documentation page for this... though the versioning of the docs themselves would become ill-defined: https://arrow.apache.org/docs/status.html ... I think it would be best to hold off on Java also, in part because of how the Java docs are integrated with the C++ and Python docs and controlled by the version selector menu. We should indeed consider how to handle the current documentation site. Previously, we actually did some work to split the sphinx docs (used for the format, dev docs, and for the Python/C++/(part of the) Java docs) into multiple sphinx projects that could be built separately (https://github.com/apache/arrow/issues/30627, https://github.com/apache/arrow/pull/11980), but we abandoned that work last year because not seeming worthwhile. But we could certainly revive that idea, for example to at least split the format docs (and let that have its own versioning based on the Format Version (currently 1.4)? or just only host a single, latest version?) Joris
Re: [DISCUSS] Versioning and releases for apache/arrow components
I am also in favor of this idea in general and in the principle, but (somewhat repeating others) I think we should be aware that this will create _more_ work overall for releasing (refactoring release scripts (at least initially), deciding which version to use for which component, etc), and not less, given that easing the burden of the release managers was mentioned as a goal. So we if pursue this, it should be for other benefits that we think this has: 1) because separate versions would be beneficial for users? (have a clearer messaging in the version number (eg no major new version if there were hardly any changes in a certain component, or use a versioning scheme more common in a certain language's ecosystem, ..?) 2) because it would actually allow separate releases, even though when initially always releasing in batch (eg being able to just do a bug fix for go, without having to also add a tag for all others) 3) because it would make the release process more manageable / easier to delegate? (and thus potentially easing the burden for an _individual_ release manager, although requiring more work overall) 4) .. other things I am missing? > We could simply release C++, R, Python and C/GLib together. > ... > > I think that versioning will require additional thinking for libraries like > > PyArrow > I think we should maybe focus on a few more obvious cases. [i.e. not C++ and > Python] Yes, agreed to not focus on those. At least for PyArrow, speaking as one of its maintainers, I am personally (at this moment) not really interested in dealing with the complexity of allowing a decoupled Arrow C++ and PyArrow build. Related to the docs: > There is a dedicated documentation page for this... though the > versioning of the docs themselves would become ill-defined: > https://arrow.apache.org/docs/status.html ... > I think it would be best to hold off on Java also, in part because > of how the Java docs are integrated with the C++ and Python docs and > controlled by the version selector menu. We should indeed consider how to handle the current documentation site. Previously, we actually did some work to split the sphinx docs (used for the format, dev docs, and for the Python/C++/(part of the) Java docs) into multiple sphinx projects that could be built separately (https://github.com/apache/arrow/issues/30627, https://github.com/apache/arrow/pull/11980), but we abandoned that work last year because not seeming worthwhile. But we could certainly revive that idea, for example to at least split the format docs (and let that have its own versioning based on the Format Version (currently 1.4)? or just only host a single, latest version?) Joris
Re: [DISCUSS] Versioning and releases for apache/arrow components
Hi David, Yeah, maybe my "wording" is not accurate, sorry about that. Core is not correct. Maybe common or another wording is more appropriate. Basically, the idea is to first release a component that it's used by other components. And the release cycle can be "atomic"/decoupled between components. Regards JB On Tue, Apr 9, 2024 at 10:44 AM David Li wrote: > > Java has JNI parts, but I think they do not necessarily need to release at > the same time as C++, especially since the JAR bundles the libraries; Java > could just pick up the latest version of the C++ library whenever it > releases. It would make it harder if the next step is to also decouple the > repositories, though. > > JB, I see what you're saying but I think we want to avoid declaring a "core" > Arrow library as the implication is not fair to the independent and > fully-featured implementations in Go, Rust, etc. But that is just a matter of > wording. > > On Tue, Apr 9, 2024, at 17:06, Jean-Baptiste Onofré wrote: > > Hi, > > > > Yeah, to be honest, I was more focused on Java versioning. > > > > Maybe, we can "group" Arrow components in two major areas: the "core" > > libs and the components using the "core" libs. > > C++ can have its own versioning, and the rest is decoupled from each > > other but it will depend to C++ release. > > > > I think it's do-able and probably "cleaner". > > > > Regards > > JB > > > > On Mon, Apr 8, 2024 at 3:55 PM Weston Pace wrote: > >> > >> > Probably major versions should match between C++ and PyArrow, but I guess > >> > we could have diverging minor and patch versions. Or at least patch > >> > versions given that > >> > a new minor version is usually cut for bug fixes too. > >> > >> I believe even this would be difficult. Stable ABIs are very finicky in > >> C++. If the public API surface changes in any way then it can lead to > >> subtle bugs if pyarrow were to link against an older version. I also am > >> not sure there is much advantage in trying to separate pyarrow from > >> arrow-cpp since they are almost always changing in lockstep (e.g. any > >> change to arrow-cpp enables functionality in pyarrow). > >> > >> I think we should maybe focus on a few more obvious cases. > >> > >> I think C#, JS, Java, and Go are the most obvious candidates to decouple. > >> Even then, we should probably only separate these candidates if they have > >> willing release managers. > >> > >> C/GLib, python, and ruby are all tightly coupled to C++ at the moment and > >> should not be a first priority. I would have guessed that R is also in > >> this list but Jacob reported in the original email that they are already > >> somewhat decoupled? > >> > >> I don't know anything about swift or matlab. > >> > >> On Mon, Apr 8, 2024 at 6:23 AM Alessandro Molina > >> wrote: > >> > >> > On Sun, Apr 7, 2024 at 3:06 PM Andrew Lamb wrote: > >> > > >> > > > >> > > We have had separate releases / votes for Arrow Rust (and Arrow > >> > DataFusion) > >> > > and it has served us quite well. The version schemes have diverged > >> > > substantially from the monorepo (we are on version 51.0.0 in arrow-rs, > >> > for > >> > > example) and it doesn't seem to have caused any large confusion with > >> > users > >> > > > >> > > > >> > I think that versioning will require additional thinking for libraries > >> > like > >> > PyArrow, Java etc... > >> > For rust this is a non problem because there is no link to the C++ > >> > library, > >> > > >> > PyArrow instead is based on what the C++ library provides, > >> > so there is a direct link between the features provided by C++ in a > >> > specific version > >> > and the features provided in PyArrow at a specific version. > >> > > >> > More or less PyArrow 20 should have the same bug fixes that C++ 20 has, > >> > and diverging the two versions would lead to confusion easily. > >> > Probably major versions should match between C++ and PyArrow, but I guess > >> > we could have diverging minor and patch versions. Or at least patch > >> > versions given that > >> > a new minor version is usually cut for bug fixes too. > >> >
Re: [DISCUSS] Versioning and releases for apache/arrow components
I think it is worthwhile to pursue this, but I fear that if we do not proceed very carefully, unforeseen complications could arise, creating even greater work for the release managers. > In general I think that this is not something we neither need to nor want to implement from 0 to 100. > Incrementally evolving and evaluating our process is key for sucsh a core change I strongly agree with this. Perhaps we could start with just one of the "easier" cases (such as C# or JS), and use that as a "pilot project" of sorts. > C/GLib, python, and ruby are all tightly coupled to C++ at the moment and should not be a first priority. I agree. I think it would be best to hold off on Java also, in part because of how the Java docs are integrated with the C++ and Python docs and controlled by the version selector menu. Ian On Tue, Apr 9, 2024 at 4:45 AM David Li wrote: > Java has JNI parts, but I think they do not necessarily need to release at > the same time as C++, especially since the JAR bundles the libraries; Java > could just pick up the latest version of the C++ library whenever it > releases. It would make it harder if the next step is to also decouple the > repositories, though. > > JB, I see what you're saying but I think we want to avoid declaring a > "core" Arrow library as the implication is not fair to the independent and > fully-featured implementations in Go, Rust, etc. But that is just a matter > of wording. > > On Tue, Apr 9, 2024, at 17:06, Jean-Baptiste Onofré wrote: > > Hi, > > > > Yeah, to be honest, I was more focused on Java versioning. > > > > Maybe, we can "group" Arrow components in two major areas: the "core" > > libs and the components using the "core" libs. > > C++ can have its own versioning, and the rest is decoupled from each > > other but it will depend to C++ release. > > > > I think it's do-able and probably "cleaner". > > > > Regards > > JB > > > > On Mon, Apr 8, 2024 at 3:55 PM Weston Pace > wrote: > >> > >> > Probably major versions should match between C++ and PyArrow, but I > guess > >> > we could have diverging minor and patch versions. Or at least patch > >> > versions given that > >> > a new minor version is usually cut for bug fixes too. > >> > >> I believe even this would be difficult. Stable ABIs are very finicky in > >> C++. If the public API surface changes in any way then it can lead to > >> subtle bugs if pyarrow were to link against an older version. I also am > >> not sure there is much advantage in trying to separate pyarrow from > >> arrow-cpp since they are almost always changing in lockstep (e.g. any > >> change to arrow-cpp enables functionality in pyarrow). > >> > >> I think we should maybe focus on a few more obvious cases. > >> > >> I think C#, JS, Java, and Go are the most obvious candidates to > decouple. > >> Even then, we should probably only separate these candidates if they > have > >> willing release managers. > >> > >> C/GLib, python, and ruby are all tightly coupled to C++ at the moment > and > >> should not be a first priority. I would have guessed that R is also in > >> this list but Jacob reported in the original email that they are already > >> somewhat decoupled? > >> > >> I don't know anything about swift or matlab. > >> > >> On Mon, Apr 8, 2024 at 6:23 AM Alessandro Molina > >> wrote: > >> > >> > On Sun, Apr 7, 2024 at 3:06 PM Andrew Lamb > wrote: > >> > > >> > > > >> > > We have had separate releases / votes for Arrow Rust (and Arrow > >> > DataFusion) > >> > > and it has served us quite well. The version schemes have diverged > >> > > substantially from the monorepo (we are on version 51.0.0 in > arrow-rs, > >> > for > >> > > example) and it doesn't seem to have caused any large confusion with > >> > users > >> > > > >> > > > >> > I think that versioning will require additional thinking for > libraries like > >> > PyArrow, Java etc... > >> > For rust this is a non problem because there is no link to the C++ > library, > >> > > >> > PyArrow instead is based on what the C++ library provides, > >> > so there is a direct link between the features provided by C++ in a > >> > specific version > >> > and the features provided in PyArrow at a specific version. > >> > > >> > More or less PyArrow 20 should have the same bug fixes that C++ 20 > has, > >> > and diverging the two versions would lead to confusion easily. > >> > Probably major versions should match between C++ and PyArrow, but I > guess > >> > we could have diverging minor and patch versions. Or at least patch > >> > versions given that > >> > a new minor version is usually cut for bug fixes too. > >> > >
Re: [DISCUSS] Versioning and releases for apache/arrow components
Java has JNI parts, but I think they do not necessarily need to release at the same time as C++, especially since the JAR bundles the libraries; Java could just pick up the latest version of the C++ library whenever it releases. It would make it harder if the next step is to also decouple the repositories, though. JB, I see what you're saying but I think we want to avoid declaring a "core" Arrow library as the implication is not fair to the independent and fully-featured implementations in Go, Rust, etc. But that is just a matter of wording. On Tue, Apr 9, 2024, at 17:06, Jean-Baptiste Onofré wrote: > Hi, > > Yeah, to be honest, I was more focused on Java versioning. > > Maybe, we can "group" Arrow components in two major areas: the "core" > libs and the components using the "core" libs. > C++ can have its own versioning, and the rest is decoupled from each > other but it will depend to C++ release. > > I think it's do-able and probably "cleaner". > > Regards > JB > > On Mon, Apr 8, 2024 at 3:55 PM Weston Pace wrote: >> >> > Probably major versions should match between C++ and PyArrow, but I guess >> > we could have diverging minor and patch versions. Or at least patch >> > versions given that >> > a new minor version is usually cut for bug fixes too. >> >> I believe even this would be difficult. Stable ABIs are very finicky in >> C++. If the public API surface changes in any way then it can lead to >> subtle bugs if pyarrow were to link against an older version. I also am >> not sure there is much advantage in trying to separate pyarrow from >> arrow-cpp since they are almost always changing in lockstep (e.g. any >> change to arrow-cpp enables functionality in pyarrow). >> >> I think we should maybe focus on a few more obvious cases. >> >> I think C#, JS, Java, and Go are the most obvious candidates to decouple. >> Even then, we should probably only separate these candidates if they have >> willing release managers. >> >> C/GLib, python, and ruby are all tightly coupled to C++ at the moment and >> should not be a first priority. I would have guessed that R is also in >> this list but Jacob reported in the original email that they are already >> somewhat decoupled? >> >> I don't know anything about swift or matlab. >> >> On Mon, Apr 8, 2024 at 6:23 AM Alessandro Molina >> wrote: >> >> > On Sun, Apr 7, 2024 at 3:06 PM Andrew Lamb wrote: >> > >> > > >> > > We have had separate releases / votes for Arrow Rust (and Arrow >> > DataFusion) >> > > and it has served us quite well. The version schemes have diverged >> > > substantially from the monorepo (we are on version 51.0.0 in arrow-rs, >> > for >> > > example) and it doesn't seem to have caused any large confusion with >> > users >> > > >> > > >> > I think that versioning will require additional thinking for libraries like >> > PyArrow, Java etc... >> > For rust this is a non problem because there is no link to the C++ library, >> > >> > PyArrow instead is based on what the C++ library provides, >> > so there is a direct link between the features provided by C++ in a >> > specific version >> > and the features provided in PyArrow at a specific version. >> > >> > More or less PyArrow 20 should have the same bug fixes that C++ 20 has, >> > and diverging the two versions would lead to confusion easily. >> > Probably major versions should match between C++ and PyArrow, but I guess >> > we could have diverging minor and patch versions. Or at least patch >> > versions given that >> > a new minor version is usually cut for bug fixes too. >> >
Re: [DISCUSS] Versioning and releases for apache/arrow components
Hi, Yeah, to be honest, I was more focused on Java versioning. Maybe, we can "group" Arrow components in two major areas: the "core" libs and the components using the "core" libs. C++ can have its own versioning, and the rest is decoupled from each other but it will depend to C++ release. I think it's do-able and probably "cleaner". Regards JB On Mon, Apr 8, 2024 at 3:55 PM Weston Pace wrote: > > > Probably major versions should match between C++ and PyArrow, but I guess > > we could have diverging minor and patch versions. Or at least patch > > versions given that > > a new minor version is usually cut for bug fixes too. > > I believe even this would be difficult. Stable ABIs are very finicky in > C++. If the public API surface changes in any way then it can lead to > subtle bugs if pyarrow were to link against an older version. I also am > not sure there is much advantage in trying to separate pyarrow from > arrow-cpp since they are almost always changing in lockstep (e.g. any > change to arrow-cpp enables functionality in pyarrow). > > I think we should maybe focus on a few more obvious cases. > > I think C#, JS, Java, and Go are the most obvious candidates to decouple. > Even then, we should probably only separate these candidates if they have > willing release managers. > > C/GLib, python, and ruby are all tightly coupled to C++ at the moment and > should not be a first priority. I would have guessed that R is also in > this list but Jacob reported in the original email that they are already > somewhat decoupled? > > I don't know anything about swift or matlab. > > On Mon, Apr 8, 2024 at 6:23 AM Alessandro Molina > wrote: > > > On Sun, Apr 7, 2024 at 3:06 PM Andrew Lamb wrote: > > > > > > > > We have had separate releases / votes for Arrow Rust (and Arrow > > DataFusion) > > > and it has served us quite well. The version schemes have diverged > > > substantially from the monorepo (we are on version 51.0.0 in arrow-rs, > > for > > > example) and it doesn't seem to have caused any large confusion with > > users > > > > > > > > I think that versioning will require additional thinking for libraries like > > PyArrow, Java etc... > > For rust this is a non problem because there is no link to the C++ library, > > > > PyArrow instead is based on what the C++ library provides, > > so there is a direct link between the features provided by C++ in a > > specific version > > and the features provided in PyArrow at a specific version. > > > > More or less PyArrow 20 should have the same bug fixes that C++ 20 has, > > and diverging the two versions would lead to confusion easily. > > Probably major versions should match between C++ and PyArrow, but I guess > > we could have diverging minor and patch versions. Or at least patch > > versions given that > > a new minor version is usually cut for bug fixes too. > >
Re: [DISCUSS] Versioning and releases for apache/arrow components
Thanks for the eager discussion, great to see that we are aligned on the broad strokes! In general I think that this is not something we neither need to nor want to implement from 0 to 100. Incrementally evolving and evaluating our process is key for sucsh a core change > I think C#, JS, [Java], and Go are the most obvious candidates to decouple. +1 (though I am not sure that Java fits due to the JNI parts?) >Even then, we should probably only separate these candidates if they have willing release managers. Ideally we would have an active release manager for each component but I don't quite see that yet and onboarding/changing technical situation will take some time, We currently handle verification and packaging for all implementations already and given that we will keep to 'batched' releases for the foreseeable future I think determining the proper version for each component (which would be on the comitters + a conventional commit like framework + automation?) should be the first step taken in parallel with refactoring the technical release infrastructure to handle different versions and partial releases. > We could simply release C++, R, Python and C/GLib together. While as mentioned R is backwards compatible down to 13.0.0 and could be versioned separately (as its development pace does not keep up with libarrow/pyarrow at the moment), this could still be a good step in the right direction. A proper semver scheme for this group/libarrow would still be a valuable addition. The Matlab implementation is currently establishing its release process and could be considered a binding like pyarrow, as far as I understand MLTBX. Thanks, Jacob Am Mo., 8. Apr. 2024 um 15:56 Uhr schrieb Weston Pace : > > Probably major versions should match between C++ and PyArrow, but I guess > > we could have diverging minor and patch versions. Or at least patch > > versions given that > > a new minor version is usually cut for bug fixes too. > > I believe even this would be difficult. Stable ABIs are very finicky in > C++. If the public API surface changes in any way then it can lead to > subtle bugs if pyarrow were to link against an older version. I also am > not sure there is much advantage in trying to separate pyarrow from > arrow-cpp since they are almost always changing in lockstep (e.g. any > change to arrow-cpp enables functionality in pyarrow). > > I think we should maybe focus on a few more obvious cases. > > I think C#, JS, Java, and Go are the most obvious candidates to decouple. > Even then, we should probably only separate these candidates if they have > willing release managers. > > C/GLib, python, and ruby are all tightly coupled to C++ at the moment and > should not be a first priority. I would have guessed that R is also in > this list but Jacob reported in the original email that they are already > somewhat decoupled? > > I don't know anything about swift or matlab. > > On Mon, Apr 8, 2024 at 6:23 AM Alessandro Molina > wrote: > > > On Sun, Apr 7, 2024 at 3:06 PM Andrew Lamb wrote: > > > > > > > > We have had separate releases / votes for Arrow Rust (and Arrow > > DataFusion) > > > and it has served us quite well. The version schemes have diverged > > > substantially from the monorepo (we are on version 51.0.0 in arrow-rs, > > for > > > example) and it doesn't seem to have caused any large confusion with > > users > > > > > > > > I think that versioning will require additional thinking for libraries > like > > PyArrow, Java etc... > > For rust this is a non problem because there is no link to the C++ > library, > > > > PyArrow instead is based on what the C++ library provides, > > so there is a direct link between the features provided by C++ in a > > specific version > > and the features provided in PyArrow at a specific version. > > > > More or less PyArrow 20 should have the same bug fixes that C++ 20 has, > > and diverging the two versions would lead to confusion easily. > > Probably major versions should match between C++ and PyArrow, but I guess > > we could have diverging minor and patch versions. Or at least patch > > versions given that > > a new minor version is usually cut for bug fixes too. > > >
Re: [DISCUSS] Versioning and releases for apache/arrow components
> Probably major versions should match between C++ and PyArrow, but I guess > we could have diverging minor and patch versions. Or at least patch > versions given that > a new minor version is usually cut for bug fixes too. I believe even this would be difficult. Stable ABIs are very finicky in C++. If the public API surface changes in any way then it can lead to subtle bugs if pyarrow were to link against an older version. I also am not sure there is much advantage in trying to separate pyarrow from arrow-cpp since they are almost always changing in lockstep (e.g. any change to arrow-cpp enables functionality in pyarrow). I think we should maybe focus on a few more obvious cases. I think C#, JS, Java, and Go are the most obvious candidates to decouple. Even then, we should probably only separate these candidates if they have willing release managers. C/GLib, python, and ruby are all tightly coupled to C++ at the moment and should not be a first priority. I would have guessed that R is also in this list but Jacob reported in the original email that they are already somewhat decoupled? I don't know anything about swift or matlab. On Mon, Apr 8, 2024 at 6:23 AM Alessandro Molina wrote: > On Sun, Apr 7, 2024 at 3:06 PM Andrew Lamb wrote: > > > > > We have had separate releases / votes for Arrow Rust (and Arrow > DataFusion) > > and it has served us quite well. The version schemes have diverged > > substantially from the monorepo (we are on version 51.0.0 in arrow-rs, > for > > example) and it doesn't seem to have caused any large confusion with > users > > > > > I think that versioning will require additional thinking for libraries like > PyArrow, Java etc... > For rust this is a non problem because there is no link to the C++ library, > > PyArrow instead is based on what the C++ library provides, > so there is a direct link between the features provided by C++ in a > specific version > and the features provided in PyArrow at a specific version. > > More or less PyArrow 20 should have the same bug fixes that C++ 20 has, > and diverging the two versions would lead to confusion easily. > Probably major versions should match between C++ and PyArrow, but I guess > we could have diverging minor and patch versions. Or at least patch > versions given that > a new minor version is usually cut for bug fixes too. >
Re: [DISCUSS] Versioning and releases for apache/arrow components
On Sun, Apr 7, 2024 at 3:06 PM Andrew Lamb wrote: > > We have had separate releases / votes for Arrow Rust (and Arrow DataFusion) > and it has served us quite well. The version schemes have diverged > substantially from the monorepo (we are on version 51.0.0 in arrow-rs, for > example) and it doesn't seem to have caused any large confusion with users > > I think that versioning will require additional thinking for libraries like PyArrow, Java etc... For rust this is a non problem because there is no link to the C++ library, PyArrow instead is based on what the C++ library provides, so there is a direct link between the features provided by C++ in a specific version and the features provided in PyArrow at a specific version. More or less PyArrow 20 should have the same bug fixes that C++ 20 has, and diverging the two versions would lead to confusion easily. Probably major versions should match between C++ and PyArrow, but I guess we could have diverging minor and patch versions. Or at least patch versions given that a new minor version is usually cut for bug fixes too.
Re: [DISCUSS] Versioning and releases for apache/arrow components
Hi, I agree with Jacob's proposal, and also share the concern with Kou. IMHO, it would be great to have several potential release managers. On some other Apache projects, we had a kind of release bottleneck with a very limited number of release managers. Maybe it would be interesting to have reviewers.yaml per component and the two first reviewers would be the "first release manager). Happy to help David on Java component if needed ;) Another discussion (not in this thread) that it would be great to have is about release versioning. I'm a bit surprised that a lot of Arrow releases use major version whereas it doesn't contain any breaking change. For our users, it would be useful if a release really include breaking change (public API changes, spec changes, ...) or just fixes (a kind of semantic versioning). But again, that's another discussion :) Regards JB On Mon, Apr 8, 2024 at 5:20 AM Sutou Kouhei wrote: > > Hi, > > I don't object this but I'm worry about we have enough > release managers for each component. Here are components: > > * C/GLib (me but I want to release C++ and C/GLib together > for .deb/.rpm packages) > * C++ (Raúl and me) > * C# (Curt?) > * Go (Matt?) > * Java (David?) > * JavaScript (Dominik?) > * MATLAB (Kevin?) > * Python (Alenka?) > * R (Jacob?) > * Ruby (me) > * Swift (me...?) > > Thanks, > -- > kou > > In > "[DISCUSS] Versioning and releases for apache/arrow components" on Thu, 28 > Mar 2024 21:42:14 +0100, > Jacob Wujciak wrote: > > > Hello Everyone! > > > > I would like to resurface the discussion of separate > > versioning/releases/voting for monorepo components. We have previously > > touched on this topic mostly in the community meetings and spread across > > multiple, only tangential related threads. I think a focused discussion can > > be a bit more results oriented, especially now that we almost regularly > > deviate from the quarterly release cadence with minor releases. My hope is > > that discussing this and adapting our process can lower the amount of work > > required and ease the pressure on our release managers (Thank you Raúl and > > Kou!). > > > > I think the base of the topic is the separate versioning for components as > > otherwise separate releases only have limited value. From a technical > > perspective standalone implementations like Go or JS are the easiest to > > handle in that regard, they can just follow their ecosystem standards, > > which has been requested by users already (major releases in Go require > > manual editing across a code base as dependencies are usually pinned to a > > major version). > > > > For Arrow C++ bindings like Arrow R and PyArrow having distinct versions > > would require additional work to both enable the use of different versions > > and ensure version compatibility is monitored and potentially updated if > > needed. > > > > For Arrow R we have already implemented these changes for different reasons > > and have backwards compatibility with libarrow >= 13.0.0. From a user > > standpoint of PyArrow this is likely irrelevant as most users get binary > > wheels from pypi, if a user regularly builds PyArrow from source they are > > also capable of managing potentially different libarrow version > > requirements as this is already necessary to build the package just with an > > exact version match. > > > > A more meta question is about the messaging that different versioning > > schemes carry, as it might no longer be obvious on first glance which > > versions are compatible or have the newest features. Though I would argue > > that this a marginal concern at best as there is no guarantee of feature > > parity between different components with the same version. Breaking that > > implicit expectation with separate versions could be seen as clearer. If a > > component only receives dependency bumps or minor bug fixes, releasing this > > component with a patch version aligns much better with expectations than a > > major version bump. In addition there are already several differently > > versioned libraries in the apache/arrow-* ecosystem that are released > > outside of the monorepo release process. A proper support policy for each > > component would also be required but could just default to 'current major > > release' as it is now. > > > > From an ASF perspective there is no requirement to release the entire > > repository at once as the actual release artifact is the source tarball. As > > long as that is verified and voted on by the PMC it is an official release. > > > > This brings me to the release process and voting. I think it is pretty > > clear that completely decoupling all components and their release processes > > isn't feasible at the moment, mainly from a technical perspective > > (crossbow) and would likely also lead to vote fatigue. We have made efforts > > to ease the verification required for the vote easier and will continue > > these efforts. Though I can see some of the components managing
Re: [DISCUSS] Versioning and releases for apache/arrow components
Hi, I don't object this but I'm worry about we have enough release managers for each component. Here are components: * C/GLib (me but I want to release C++ and C/GLib together for .deb/.rpm packages) * C++ (Raúl and me) * C# (Curt?) * Go (Matt?) * Java (David?) * JavaScript (Dominik?) * MATLAB (Kevin?) * Python (Alenka?) * R (Jacob?) * Ruby (me) * Swift (me...?) Thanks, -- kou In "[DISCUSS] Versioning and releases for apache/arrow components" on Thu, 28 Mar 2024 21:42:14 +0100, Jacob Wujciak wrote: > Hello Everyone! > > I would like to resurface the discussion of separate > versioning/releases/voting for monorepo components. We have previously > touched on this topic mostly in the community meetings and spread across > multiple, only tangential related threads. I think a focused discussion can > be a bit more results oriented, especially now that we almost regularly > deviate from the quarterly release cadence with minor releases. My hope is > that discussing this and adapting our process can lower the amount of work > required and ease the pressure on our release managers (Thank you Raúl and > Kou!). > > I think the base of the topic is the separate versioning for components as > otherwise separate releases only have limited value. From a technical > perspective standalone implementations like Go or JS are the easiest to > handle in that regard, they can just follow their ecosystem standards, > which has been requested by users already (major releases in Go require > manual editing across a code base as dependencies are usually pinned to a > major version). > > For Arrow C++ bindings like Arrow R and PyArrow having distinct versions > would require additional work to both enable the use of different versions > and ensure version compatibility is monitored and potentially updated if > needed. > > For Arrow R we have already implemented these changes for different reasons > and have backwards compatibility with libarrow >= 13.0.0. From a user > standpoint of PyArrow this is likely irrelevant as most users get binary > wheels from pypi, if a user regularly builds PyArrow from source they are > also capable of managing potentially different libarrow version > requirements as this is already necessary to build the package just with an > exact version match. > > A more meta question is about the messaging that different versioning > schemes carry, as it might no longer be obvious on first glance which > versions are compatible or have the newest features. Though I would argue > that this a marginal concern at best as there is no guarantee of feature > parity between different components with the same version. Breaking that > implicit expectation with separate versions could be seen as clearer. If a > component only receives dependency bumps or minor bug fixes, releasing this > component with a patch version aligns much better with expectations than a > major version bump. In addition there are already several differently > versioned libraries in the apache/arrow-* ecosystem that are released > outside of the monorepo release process. A proper support policy for each > component would also be required but could just default to 'current major > release' as it is now. > > From an ASF perspective there is no requirement to release the entire > repository at once as the actual release artifact is the source tarball. As > long as that is verified and voted on by the PMC it is an official release. > > This brings me to the release process and voting. I think it is pretty > clear that completely decoupling all components and their release processes > isn't feasible at the moment, mainly from a technical perspective > (crossbow) and would likely also lead to vote fatigue. We have made efforts > to ease the verification required for the vote easier and will continue > these efforts. Though I can see some of the components managing their own > releases (e.g. R, as we do with post release tasks already due to CRAN, ) a > continued quarterly 'batch release' seems like a more appealing solution > and would still allow us to use separate versions. Voting in one thread on > all components/a subset of components per voter and the surrounding > technicalities is something I would like to hear some opinions on. > > In my opinion being stricter with release requirements for components might > lead to smaller/less active components not releasing. This seems like a > bad thing at first glance but might also spur the user community to get > involved when the reassuring, regular releases dry up and reflect the > reality of the development situation of the component. > > I am eager to hear your thoughts! > > Best > Jacob
Re: [DISCUSS] Versioning and releases for apache/arrow components
Le 28/03/2024 à 21:42, Jacob Wujciak a écrit : For Arrow C++ bindings like Arrow R and PyArrow having distinct versions would require additional work to both enable the use of different versions and ensure version compatibility is monitored and potentially updated if needed. We could simply release C++, R, Python and C/GLib together. A more meta question is about the messaging that different versioning schemes carry, as it might no longer be obvious on first glance which versions are compatible or have the newest features. There is a dedicated documentation page for this... though the versioning of the docs themselves would become ill-defined: https://arrow.apache.org/docs/status.html Regards Antoine.
Re: [DISCUSS] Versioning and releases for apache/arrow components
I agree with all the other comments on this thread Having smaller releases is key to being able to release more frequently and finding the relevant expertise in my opinion. We have had separate releases / votes for Arrow Rust (and Arrow DataFusion) and it has served us quite well. The version schemes have diverged substantially from the monorepo (we are on version 51.0.0 in arrow-rs, for example) and it doesn't seem to have caused any large confusion with users Andrew On Wed, Apr 3, 2024 at 2:11 PM Dewey Dunnington wrote: > Thank you Jacob for bringing this up! I am also in favor of decoupling > versions (provided that the release managers are also in favor of > this, since their time is required to implement this and because the > ongoing consequences of separate releases disproportionately affects > them). > > Part of the vote fatigue is, I think, partly due to the complexity of > releasing all of the components at the same time. Running the script > for ADBC, nanoarrow, Rust, and Julia are all fairly straightforward > because those subprojects have a more limited scope. In contrast, I am > rarely successful running the Arrow verification script without > running into an error I don't understand and have become hesitant to > vote (or try) as a cumulative result of many releases worth of this > happening (and because R has never been a part of verification, which > is the component that I unofficially verify anyway). Voting on a batch > of version numbers seems like a good first step. > > I am also not concerned about messaging of different versions of > different components. The fact that integration tests pass at the > moment of the release may be meaningful for those familiar with the > repo, but I don't think that many people are aware of which components > are tested in that way. As Weston noted, even for components that use > Arrow C++, the implementation of Arrow C++ features may lag behind or > be completely unrelated (Python being the exception). > > On Fri, Mar 29, 2024 at 9:47 AM Weston Pace wrote: > > > > Thank you for bringing this up. I'm in favor of this. I think there are > > several motivations but the main ones are: > > > > 1. Decoupling the versions will allow components to have no release, or > > only a minor release, when there are no breaking changes > > 2. We do have some vote fatigue I think and we don't want to make that > > more difficult. > > 3. Anything we can do to ease the burden of release managers is good > > > > If I understand what you are describing then I think it satisfies points > 1 > > & 2. I am not familiar enough with the release management process to > speak > > to #3. > > > > > Voting in one thread on > > > all components/a subset of components per voter and the surrounding > > > technicalities is something I would like to hear some opinions on. > > > > I am in favor of decoupling the version numbers. I do think batched > > quarterly releases are still a good thing to avoid vote fatigue. Perhaps > > we can have a single vote on a batch of version numbers (e.g. please vote > > on the batched release containing CPP version X, Go version Y, JS version > > Z). > > > > > A more meta question is about the messaging that different versioning > > > schemes carry, as it might no longer be obvious on first glance which > > > versions are compatible or have the newest features. > > > > I am not concerned about this. One of the advantages of Arrow is that we > > have a stable C ABI (C Data Interface) and a stable IPC mechanism (IPC > > serialization) and this means that version compatibility is rarely a > > difficulty or major concern. Plus, regarding individual features, our > > solution already requires a compatibility table ( > > https://arrow.apache.org/docs/status.html). Changing the versioning > > strategy will not make this any worse. > > > > On Thu, Mar 28, 2024 at 1:42 PM Jacob Wujciak > wrote: > > > > > Hello Everyone! > > > > > > I would like to resurface the discussion of separate > > > versioning/releases/voting for monorepo components. We have previously > > > touched on this topic mostly in the community meetings and spread > across > > > multiple, only tangential related threads. I think a focused > discussion can > > > be a bit more results oriented, especially now that we almost regularly > > > deviate from the quarterly release cadence with minor releases. My > hope is > > > that discussing this and adapting our process can lower the amount of > work > > > required and ease the pressure on our release managers (Thank you Raúl > and > > > Kou!). > > > > > > I think the base of the topic is the separate versioning for > components as > > > otherwise separate releases only have limited value. From a technical > > > perspective standalone implementations like Go or JS are the easiest to > > > handle in that regard, they can just follow their ecosystem standards, > > > which has been requested by users already (major releases in Go require
Re: [DISCUSS] Versioning and releases for apache/arrow components
Thank you Jacob for bringing this up! I am also in favor of decoupling versions (provided that the release managers are also in favor of this, since their time is required to implement this and because the ongoing consequences of separate releases disproportionately affects them). Part of the vote fatigue is, I think, partly due to the complexity of releasing all of the components at the same time. Running the script for ADBC, nanoarrow, Rust, and Julia are all fairly straightforward because those subprojects have a more limited scope. In contrast, I am rarely successful running the Arrow verification script without running into an error I don't understand and have become hesitant to vote (or try) as a cumulative result of many releases worth of this happening (and because R has never been a part of verification, which is the component that I unofficially verify anyway). Voting on a batch of version numbers seems like a good first step. I am also not concerned about messaging of different versions of different components. The fact that integration tests pass at the moment of the release may be meaningful for those familiar with the repo, but I don't think that many people are aware of which components are tested in that way. As Weston noted, even for components that use Arrow C++, the implementation of Arrow C++ features may lag behind or be completely unrelated (Python being the exception). On Fri, Mar 29, 2024 at 9:47 AM Weston Pace wrote: > > Thank you for bringing this up. I'm in favor of this. I think there are > several motivations but the main ones are: > > 1. Decoupling the versions will allow components to have no release, or > only a minor release, when there are no breaking changes > 2. We do have some vote fatigue I think and we don't want to make that > more difficult. > 3. Anything we can do to ease the burden of release managers is good > > If I understand what you are describing then I think it satisfies points 1 > & 2. I am not familiar enough with the release management process to speak > to #3. > > > Voting in one thread on > > all components/a subset of components per voter and the surrounding > > technicalities is something I would like to hear some opinions on. > > I am in favor of decoupling the version numbers. I do think batched > quarterly releases are still a good thing to avoid vote fatigue. Perhaps > we can have a single vote on a batch of version numbers (e.g. please vote > on the batched release containing CPP version X, Go version Y, JS version > Z). > > > A more meta question is about the messaging that different versioning > > schemes carry, as it might no longer be obvious on first glance which > > versions are compatible or have the newest features. > > I am not concerned about this. One of the advantages of Arrow is that we > have a stable C ABI (C Data Interface) and a stable IPC mechanism (IPC > serialization) and this means that version compatibility is rarely a > difficulty or major concern. Plus, regarding individual features, our > solution already requires a compatibility table ( > https://arrow.apache.org/docs/status.html). Changing the versioning > strategy will not make this any worse. > > On Thu, Mar 28, 2024 at 1:42 PM Jacob Wujciak wrote: > > > Hello Everyone! > > > > I would like to resurface the discussion of separate > > versioning/releases/voting for monorepo components. We have previously > > touched on this topic mostly in the community meetings and spread across > > multiple, only tangential related threads. I think a focused discussion can > > be a bit more results oriented, especially now that we almost regularly > > deviate from the quarterly release cadence with minor releases. My hope is > > that discussing this and adapting our process can lower the amount of work > > required and ease the pressure on our release managers (Thank you Raúl and > > Kou!). > > > > I think the base of the topic is the separate versioning for components as > > otherwise separate releases only have limited value. From a technical > > perspective standalone implementations like Go or JS are the easiest to > > handle in that regard, they can just follow their ecosystem standards, > > which has been requested by users already (major releases in Go require > > manual editing across a code base as dependencies are usually pinned to a > > major version). > > > > For Arrow C++ bindings like Arrow R and PyArrow having distinct versions > > would require additional work to both enable the use of different versions > > and ensure version compatibility is monitored and potentially updated if > > needed. > > > > For Arrow R we have already implemented these changes for different reasons > > and have backwards compatibility with libarrow >= 13.0.0. From a user > > standpoint of PyArrow this is likely irrelevant as most users get binary > > wheels from pypi, if a user regularly builds PyArrow from source they are > > also capable of managing potentially different
Re: [DISCUSS] Versioning and releases for apache/arrow components
Thank you for bringing this up. I'm in favor of this. I think there are several motivations but the main ones are: 1. Decoupling the versions will allow components to have no release, or only a minor release, when there are no breaking changes 2. We do have some vote fatigue I think and we don't want to make that more difficult. 3. Anything we can do to ease the burden of release managers is good If I understand what you are describing then I think it satisfies points 1 & 2. I am not familiar enough with the release management process to speak to #3. > Voting in one thread on > all components/a subset of components per voter and the surrounding > technicalities is something I would like to hear some opinions on. I am in favor of decoupling the version numbers. I do think batched quarterly releases are still a good thing to avoid vote fatigue. Perhaps we can have a single vote on a batch of version numbers (e.g. please vote on the batched release containing CPP version X, Go version Y, JS version Z). > A more meta question is about the messaging that different versioning > schemes carry, as it might no longer be obvious on first glance which > versions are compatible or have the newest features. I am not concerned about this. One of the advantages of Arrow is that we have a stable C ABI (C Data Interface) and a stable IPC mechanism (IPC serialization) and this means that version compatibility is rarely a difficulty or major concern. Plus, regarding individual features, our solution already requires a compatibility table ( https://arrow.apache.org/docs/status.html). Changing the versioning strategy will not make this any worse. On Thu, Mar 28, 2024 at 1:42 PM Jacob Wujciak wrote: > Hello Everyone! > > I would like to resurface the discussion of separate > versioning/releases/voting for monorepo components. We have previously > touched on this topic mostly in the community meetings and spread across > multiple, only tangential related threads. I think a focused discussion can > be a bit more results oriented, especially now that we almost regularly > deviate from the quarterly release cadence with minor releases. My hope is > that discussing this and adapting our process can lower the amount of work > required and ease the pressure on our release managers (Thank you Raúl and > Kou!). > > I think the base of the topic is the separate versioning for components as > otherwise separate releases only have limited value. From a technical > perspective standalone implementations like Go or JS are the easiest to > handle in that regard, they can just follow their ecosystem standards, > which has been requested by users already (major releases in Go require > manual editing across a code base as dependencies are usually pinned to a > major version). > > For Arrow C++ bindings like Arrow R and PyArrow having distinct versions > would require additional work to both enable the use of different versions > and ensure version compatibility is monitored and potentially updated if > needed. > > For Arrow R we have already implemented these changes for different reasons > and have backwards compatibility with libarrow >= 13.0.0. From a user > standpoint of PyArrow this is likely irrelevant as most users get binary > wheels from pypi, if a user regularly builds PyArrow from source they are > also capable of managing potentially different libarrow version > requirements as this is already necessary to build the package just with an > exact version match. > > A more meta question is about the messaging that different versioning > schemes carry, as it might no longer be obvious on first glance which > versions are compatible or have the newest features. Though I would argue > that this a marginal concern at best as there is no guarantee of feature > parity between different components with the same version. Breaking that > implicit expectation with separate versions could be seen as clearer. If a > component only receives dependency bumps or minor bug fixes, releasing this > component with a patch version aligns much better with expectations than a > major version bump. In addition there are already several differently > versioned libraries in the apache/arrow-* ecosystem that are released > outside of the monorepo release process. A proper support policy for each > component would also be required but could just default to 'current major > release' as it is now. > > From an ASF perspective there is no requirement to release the entire > repository at once as the actual release artifact is the source tarball. As > long as that is verified and voted on by the PMC it is an official release. > > This brings me to the release process and voting. I think it is pretty > clear that completely decoupling all components and their release processes > isn't feasible at the moment, mainly from a technical perspective > (crossbow) and would likely also lead to vote fatigue. We have made efforts > to ease the
