I personally believe we should continue to release the rust arrow crate in such a way that the major versions match the other implementations, for precisely the reasons you mention.
On Fri, Apr 9, 2021 at 4:34 PM Neal Richardson <neal.p.richard...@gmail.com> wrote: > Thanks, Andy. Two areas of concern I think we should have some answer for > before going forward with this (and I make no opinions as to what the > "right" answers are, just raising them for discussion): > > 1. Integration testing: what is our workflow for ensuring that our > implementations are integration tested, and what do we do when changes > (whether in apache/arrow or in apache/arrow-rs) introduce > regressions/failures? I'm assuming the idea is that the existing > integration tests will remain in apache/arrow. Will you also run the > integration test suites on your rust repository CI checks? > 2. Versioning: one rationale from our current policy of "everyone releases > together" is that you don't have to guess as much whether (for example) > Arrow Java 3.0 and Arrow Rust 3.0 are compatible and using the same format. > It's kind of a heuristic for what library versions were integration tested > with each other. It sounds like (but maybe I misunderstand) that y'all are > looking to break from that. But if Arrow C++ goes to version 7.0 by the end > of the year and arrow-rs chooses to go to 15.4, or 3.12, or whatever, does > that create confusion or doubt that works against the Arrow goal of easy > interoperability? > > Neal > > On Fri, Apr 9, 2021 at 8:18 AM Andy Grove <andygrov...@gmail.com> wrote: > > > Following on from the email thread "Rust sync meeting" I would like to > > start a new discussion about moving the Rust components out to new GitHub > > repositories and using a new process for issues and release management. > > > > I have started a Google document [1] with details and to track the work > > required for this effort but I will summarize the key points of the > > proposal here: > > > > > > - > > > > Move existing Rust code into two new repositories > > - > > > > apache/arrow-rs > > - > > > > Arrow + Parquet crates > > - > > > > apache/datafusion > > - > > > > DataFusion + Ballista crates (which are expected to merge to > some > > degree over time) > > - > > > > TPC-H benchmarks > > - > > > > Use GitHub issues for issue tracking > > - > > > > Decouple release process > > - > > > > Crates are released individually > > - > > > > A vote on the source release of the released crate is held over the > > mailing list as usual. > > - > > > > Rust does not need to release a new version when the rest of Arrow > > releases; we bundle our latest released crates to the signed tar. > > - > > > > Crates can depend on GitHub commit hashes between releases > > > > > > The Google document may be the best place to collaborate on the proposal > > but I can update the document based on any comments in this email thread > as > > well. > > > > Note that I have excluded discussion about arrow2/parquet2 from this > > proposal and I believe we should discuss that separately as a follow-on > > discussion. > > > > I look forward to hearing opinions on this both from current Rust > > maintainers and contributors and also from the wider Arrow community. > > > > Thanks, > > > > Andy. > > > > [1] > > > > > https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit?usp=sharing > > >
