Re: Aapche process questions

[Wes McKinney]

Yes, that is correct. We don't want to create confusion between releases
made by the Apache project and unofficial development releases

On Wed, Apr 4, 2018, 10:17 PM Andy Grove  wrote:

> Wes,
>
> We talked about nightlies and downstream dependencies and I just want to be
> clear I'm understanding what you are suggesting. Apologies if I'm being a
> bit slow.
>
> I think we are in agreement that nightlies make sense for now because
> things are moving fast and this Rust library is very new.
>
> I can build nightly releases myself, but if I publish those to crates.io,
> they should be under a separate name and not using the "arrow" crate that I
> established for this project on crates.io
>
> Is that correct?
>
> Thanks,
>
> Andy.
>
> On Wed, Apr 4, 2018 at 6:29 PM, Wes McKinney  wrote:
>
> > Hi Andy,
> >
> > You're free to do whatever you like outside of Apache infrastructure -- I
> > don't think we can have the apache/arrow Travis publishing artifacts to
> > crates.io, though.
> >
> > If you want to have some scripts in the repo to help with creating
> > nightlies, that's totally fine, but this isn't something that the PMC is
> > able to do anything about formally. We documented our nightly build tool
> > for Python here
> >
> > https://github.com/apache/arrow/blob/master/python/doc/
> > source/development.rst#nightly-builds-of-arrow-cpp-
> > parquet-cpp-and-pyarrow-for-linux
> >
> > - Wes
> >
> > On Wed, Apr 4, 2018, 7:13 PM Andy Grove  wrote:
> >
> > > Wes,
> > >
> > > Thanks for the feedback.
> > >
> > > A nightly release sounds appealing in these early days. Technically, I
> > > assume this is just a case of configuring travis to update the minor
> > > version number and run a "cargo publish" on master once per day (if
> there
> > > are changes)?
> > >
> > > From a process point of view, what is involved in deciding whether to
> do
> > > this or not?
> > >
> > > Thanks,
> > >
> > > Andy.
> > >
> > >
> > >
> > >
> > > On Wed, Apr 4, 2018 at 8:18 AM, Wes McKinney 
> > wrote:
> > >
> > > > hi Andy,
> > > >
> > > > > My argument for doing this is that although other Rust developers
> can
> > > > set up a github dependency in their Cargo,toml, this just isn't a
> > natural
> > > > way to work in Rust so we are making it hard for people to experiment
> > > with
> > > > Arrow.
> > > >
> > > > Being penalized by the language for not using a centralized package
> > > > repository seems odd to me, but if that's the way Rust is, then so be
> > > > it.
> > > >
> > > > > 1. Is it ok for me to push updated releases to crates.io?
> > > >
> > > > If crates.io definitely needs to get updated as frequently as you
> are
> > > > saying, we should make separate Rust releases and hold votes like we
> > > > have with JavaScript. It sounds like nightlies or downstream
> > > > "releases" would be a better solution while the project is new / in
> > > > flux, see below:
> > > >
> > > > > 2. Is releasing from a fork of the code (under a different name)
> > > > acceptable?
> > > >
> > > > Sure, you can make your own "downstream" releases of the project, as
> > > > long as they aren't advertised as being releases made by the Apache
> > > > project. Some of us have been building nightly Python packages for
> > > > development purposes
> > > >
> > > > - Wes
> > > >
> > > > On Wed, Apr 4, 2018 at 10:01 AM, Andy Grove 
> > > wrote:
> > > > > Hi,
> > > > >
> > > > > I've been creating some frustrations for myself this week because
> I'm
> > > not
> > > > > sure how to work efficiently now that the Rust version of Apache
> > Arrow
> > > is
> > > > > in the official Apache repo.
> > > > >
> > > > > It seems I have two conflicting requirements:
> > > > >
> > > > > 1. I want Apache Arrow [Rust] to be a community-driven high quality
> > > piece
> > > > > of software built by many contributors, with a thoughtful code
> review
> > > > > process.
> > > > >
> > > > > 2. I want to move fast and innovate on my other open source project
> > > which
> > > > > now depends on Arrow, and I want to help other projects (such as
> > > > > https://github.com/sunchao/parquet-rs) integrate with Arrow.
> > > > >
> > > > > I have two questions that I need guidance on:
> > > > >
> > > > > 1. Is it ok for me to push updated releases to crates.io?
> > > > >
> > > > > We currently have a 0.1.0 release of the Rust library in crates.io
> (
> > > > > https://crates.io/crates/arrow) which was made to reserve the
> Arrow
> > > name
> > > > > there (with approval from Wes). The github repo has changed quite a
> > bit
> > > > > since this release but this is the only version that Rust users can
> > > > easily
> > > > > pull into their projects as a versioned dependency. Understanding
> > that
> > > > this
> > > > > isn't an official Apache release since it hasn't been through a
> > release
> > > > > process, is it OK to push new versions of this unofficial 

Re: Aapche process questions

[Andy Grove]

Wes,

We talked about nightlies and downstream dependencies and I just want to be
clear I'm understanding what you are suggesting. Apologies if I'm being a
bit slow.

I think we are in agreement that nightlies make sense for now because
things are moving fast and this Rust library is very new.

I can build nightly releases myself, but if I publish those to crates.io,
they should be under a separate name and not using the "arrow" crate that I
established for this project on crates.io

Is that correct?

Thanks,

Andy.

On Wed, Apr 4, 2018 at 6:29 PM, Wes McKinney  wrote:

> Hi Andy,
>
> You're free to do whatever you like outside of Apache infrastructure -- I
> don't think we can have the apache/arrow Travis publishing artifacts to
> crates.io, though.
>
> If you want to have some scripts in the repo to help with creating
> nightlies, that's totally fine, but this isn't something that the PMC is
> able to do anything about formally. We documented our nightly build tool
> for Python here
>
> https://github.com/apache/arrow/blob/master/python/doc/
> source/development.rst#nightly-builds-of-arrow-cpp-
> parquet-cpp-and-pyarrow-for-linux
>
> - Wes
>
> On Wed, Apr 4, 2018, 7:13 PM Andy Grove  wrote:
>
> > Wes,
> >
> > Thanks for the feedback.
> >
> > A nightly release sounds appealing in these early days. Technically, I
> > assume this is just a case of configuring travis to update the minor
> > version number and run a "cargo publish" on master once per day (if there
> > are changes)?
> >
> > From a process point of view, what is involved in deciding whether to do
> > this or not?
> >
> > Thanks,
> >
> > Andy.
> >
> >
> >
> >
> > On Wed, Apr 4, 2018 at 8:18 AM, Wes McKinney 
> wrote:
> >
> > > hi Andy,
> > >
> > > > My argument for doing this is that although other Rust developers can
> > > set up a github dependency in their Cargo,toml, this just isn't a
> natural
> > > way to work in Rust so we are making it hard for people to experiment
> > with
> > > Arrow.
> > >
> > > Being penalized by the language for not using a centralized package
> > > repository seems odd to me, but if that's the way Rust is, then so be
> > > it.
> > >
> > > > 1. Is it ok for me to push updated releases to crates.io?
> > >
> > > If crates.io definitely needs to get updated as frequently as you are
> > > saying, we should make separate Rust releases and hold votes like we
> > > have with JavaScript. It sounds like nightlies or downstream
> > > "releases" would be a better solution while the project is new / in
> > > flux, see below:
> > >
> > > > 2. Is releasing from a fork of the code (under a different name)
> > > acceptable?
> > >
> > > Sure, you can make your own "downstream" releases of the project, as
> > > long as they aren't advertised as being releases made by the Apache
> > > project. Some of us have been building nightly Python packages for
> > > development purposes
> > >
> > > - Wes
> > >
> > > On Wed, Apr 4, 2018 at 10:01 AM, Andy Grove 
> > wrote:
> > > > Hi,
> > > >
> > > > I've been creating some frustrations for myself this week because I'm
> > not
> > > > sure how to work efficiently now that the Rust version of Apache
> Arrow
> > is
> > > > in the official Apache repo.
> > > >
> > > > It seems I have two conflicting requirements:
> > > >
> > > > 1. I want Apache Arrow [Rust] to be a community-driven high quality
> > piece
> > > > of software built by many contributors, with a thoughtful code review
> > > > process.
> > > >
> > > > 2. I want to move fast and innovate on my other open source project
> > which
> > > > now depends on Arrow, and I want to help other projects (such as
> > > > https://github.com/sunchao/parquet-rs) integrate with Arrow.
> > > >
> > > > I have two questions that I need guidance on:
> > > >
> > > > 1. Is it ok for me to push updated releases to crates.io?
> > > >
> > > > We currently have a 0.1.0 release of the Rust library in crates.io (
> > > > https://crates.io/crates/arrow) which was made to reserve the Arrow
> > name
> > > > there (with approval from Wes). The github repo has changed quite a
> bit
> > > > since this release but this is the only version that Rust users can
> > > easily
> > > > pull into their projects as a versioned dependency. Understanding
> that
> > > this
> > > > isn't an official Apache release since it hasn't been through a
> release
> > > > process, is it OK to push new versions of this unofficial release as
> > PRs
> > > > are accepted into the repo? I have the *ability* to do this but I
> don't
> > > > know if I have *approval* to do this.
> > > >
> > > > My argument for doing this is that although other Rust developers can
> > set
> > > > up a github dependency in their Cargo,toml, this just isn't a natural
> > way
> > > > to work in Rust so we are making it hard for people to experiment
> with
> > > > Arrow. The code is available in github and anyone could fork the code
> 

Re: Aapche process questions

[Andy Grove]

Wes,

Thanks for the feedback.

A nightly release sounds appealing in these early days. Technically, I
assume this is just a case of configuring travis to update the minor
version number and run a "cargo publish" on master once per day (if there
are changes)?

>From a process point of view, what is involved in deciding whether to do
this or not?

Thanks,

Andy.




On Wed, Apr 4, 2018 at 8:18 AM, Wes McKinney  wrote:

> hi Andy,
>
> > My argument for doing this is that although other Rust developers can
> set up a github dependency in their Cargo,toml, this just isn't a natural
> way to work in Rust so we are making it hard for people to experiment with
> Arrow.
>
> Being penalized by the language for not using a centralized package
> repository seems odd to me, but if that's the way Rust is, then so be
> it.
>
> > 1. Is it ok for me to push updated releases to crates.io?
>
> If crates.io definitely needs to get updated as frequently as you are
> saying, we should make separate Rust releases and hold votes like we
> have with JavaScript. It sounds like nightlies or downstream
> "releases" would be a better solution while the project is new / in
> flux, see below:
>
> > 2. Is releasing from a fork of the code (under a different name)
> acceptable?
>
> Sure, you can make your own "downstream" releases of the project, as
> long as they aren't advertised as being releases made by the Apache
> project. Some of us have been building nightly Python packages for
> development purposes
>
> - Wes
>
> On Wed, Apr 4, 2018 at 10:01 AM, Andy Grove  wrote:
> > Hi,
> >
> > I've been creating some frustrations for myself this week because I'm not
> > sure how to work efficiently now that the Rust version of Apache Arrow is
> > in the official Apache repo.
> >
> > It seems I have two conflicting requirements:
> >
> > 1. I want Apache Arrow [Rust] to be a community-driven high quality piece
> > of software built by many contributors, with a thoughtful code review
> > process.
> >
> > 2. I want to move fast and innovate on my other open source project which
> > now depends on Arrow, and I want to help other projects (such as
> > https://github.com/sunchao/parquet-rs) integrate with Arrow.
> >
> > I have two questions that I need guidance on:
> >
> > 1. Is it ok for me to push updated releases to crates.io?
> >
> > We currently have a 0.1.0 release of the Rust library in crates.io (
> > https://crates.io/crates/arrow) which was made to reserve the Arrow name
> > there (with approval from Wes). The github repo has changed quite a bit
> > since this release but this is the only version that Rust users can
> easily
> > pull into their projects as a versioned dependency. Understanding that
> this
> > isn't an official Apache release since it hasn't been through a release
> > process, is it OK to push new versions of this unofficial release as PRs
> > are accepted into the repo? I have the *ability* to do this but I don't
> > know if I have *approval* to do this.
> >
> > My argument for doing this is that although other Rust developers can set
> > up a github dependency in their Cargo,toml, this just isn't a natural way
> > to work in Rust so we are making it hard for people to experiment with
> > Arrow. The code is available in github and anyone could fork the code and
> > make their own release to crates.io but I'd prefer them to use the one
> we
> > make.
> >
> > 2. Is releasing from a fork of the code (under a different name)
> acceptable?
> >
> > We cannot stop people forking the Arrow repo and making their own
> releases
> > to crates.io under a different name.
> >
> > I'm wondering if I should just do that for now e.g. release a
> > "datafusion-arrow" project from the branch in my fork where I have merged
> > all of my PRs. This way I can keep moving fast and if other projects such
> > as parquet-rs want to depend on my releases as a temporary measure, they
> > can, until the official Arrow crate catches up. This would fix my short
> > term pains but I don't want to detract from the official project.
> >
> > I'd appreciate some guidance on the best way forward.
> >
> > Thanks,
> >
> > Andy.
>

Re: Aapche process questions

[Wes McKinney]

hi Andy,

> My argument for doing this is that although other Rust developers can set up 
> a github dependency in their Cargo,toml, this just isn't a natural way to 
> work in Rust so we are making it hard for people to experiment with Arrow.

Being penalized by the language for not using a centralized package
repository seems odd to me, but if that's the way Rust is, then so be
it.

> 1. Is it ok for me to push updated releases to crates.io?

If crates.io definitely needs to get updated as frequently as you are
saying, we should make separate Rust releases and hold votes like we
have with JavaScript. It sounds like nightlies or downstream
"releases" would be a better solution while the project is new / in
flux, see below:

> 2. Is releasing from a fork of the code (under a different name) acceptable?

Sure, you can make your own "downstream" releases of the project, as
long as they aren't advertised as being releases made by the Apache
project. Some of us have been building nightly Python packages for
development purposes

- Wes

On Wed, Apr 4, 2018 at 10:01 AM, Andy Grove  wrote:
> Hi,
>
> I've been creating some frustrations for myself this week because I'm not
> sure how to work efficiently now that the Rust version of Apache Arrow is
> in the official Apache repo.
>
> It seems I have two conflicting requirements:
>
> 1. I want Apache Arrow [Rust] to be a community-driven high quality piece
> of software built by many contributors, with a thoughtful code review
> process.
>
> 2. I want to move fast and innovate on my other open source project which
> now depends on Arrow, and I want to help other projects (such as
> https://github.com/sunchao/parquet-rs) integrate with Arrow.
>
> I have two questions that I need guidance on:
>
> 1. Is it ok for me to push updated releases to crates.io?
>
> We currently have a 0.1.0 release of the Rust library in crates.io (
> https://crates.io/crates/arrow) which was made to reserve the Arrow name
> there (with approval from Wes). The github repo has changed quite a bit
> since this release but this is the only version that Rust users can easily
> pull into their projects as a versioned dependency. Understanding that this
> isn't an official Apache release since it hasn't been through a release
> process, is it OK to push new versions of this unofficial release as PRs
> are accepted into the repo? I have the *ability* to do this but I don't
> know if I have *approval* to do this.
>
> My argument for doing this is that although other Rust developers can set
> up a github dependency in their Cargo,toml, this just isn't a natural way
> to work in Rust so we are making it hard for people to experiment with
> Arrow. The code is available in github and anyone could fork the code and
> make their own release to crates.io but I'd prefer them to use the one we
> make.
>
> 2. Is releasing from a fork of the code (under a different name) acceptable?
>
> We cannot stop people forking the Arrow repo and making their own releases
> to crates.io under a different name.
>
> I'm wondering if I should just do that for now e.g. release a
> "datafusion-arrow" project from the branch in my fork where I have merged
> all of my PRs. This way I can keep moving fast and if other projects such
> as parquet-rs want to depend on my releases as a temporary measure, they
> can, until the official Arrow crate catches up. This would fix my short
> term pains but I don't want to detract from the official project.
>
> I'd appreciate some guidance on the best way forward.
>
> Thanks,
>
> Andy.

Aapche process questions

[Andy Grove]

Hi,

I've been creating some frustrations for myself this week because I'm not
sure how to work efficiently now that the Rust version of Apache Arrow is
in the official Apache repo.

It seems I have two conflicting requirements:

1. I want Apache Arrow [Rust] to be a community-driven high quality piece
of software built by many contributors, with a thoughtful code review
process.

2. I want to move fast and innovate on my other open source project which
now depends on Arrow, and I want to help other projects (such as
https://github.com/sunchao/parquet-rs) integrate with Arrow.

I have two questions that I need guidance on:

1. Is it ok for me to push updated releases to crates.io?

We currently have a 0.1.0 release of the Rust library in crates.io (
https://crates.io/crates/arrow) which was made to reserve the Arrow name
there (with approval from Wes). The github repo has changed quite a bit
since this release but this is the only version that Rust users can easily
pull into their projects as a versioned dependency. Understanding that this
isn't an official Apache release since it hasn't been through a release
process, is it OK to push new versions of this unofficial release as PRs
are accepted into the repo? I have the *ability* to do this but I don't
know if I have *approval* to do this.

My argument for doing this is that although other Rust developers can set
up a github dependency in their Cargo,toml, this just isn't a natural way
to work in Rust so we are making it hard for people to experiment with
Arrow. The code is available in github and anyone could fork the code and
make their own release to crates.io but I'd prefer them to use the one we
make.

2. Is releasing from a fork of the code (under a different name) acceptable?

We cannot stop people forking the Arrow repo and making their own releases
to crates.io under a different name.

I'm wondering if I should just do that for now e.g. release a
"datafusion-arrow" project from the branch in my fork where I have merged
all of my PRs. This way I can keep moving fast and if other projects such
as parquet-rs want to depend on my releases as a temporary measure, they
can, until the official Arrow crate catches up. This would fix my short
term pains but I don't want to detract from the official project.

I'd appreciate some guidance on the best way forward.

Thanks,

Andy.