> Is the change to the nightly compiler the kind of thing that the PMC should > vote on?
A vote is not necessary if there is reasonable consensus. On Wed, Nov 21, 2018 at 9:11 PM paddy horan <paddyho...@hotmail.com> wrote: > > I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll take a > closer look when I get a chance. I agree that lack of support for 32 bit > should not hold this up. > > Is the change to the nightly compiler the kind of thing that the PMC should > vote on? This could be done in advance of the code donantion. > Specialization, in particular, would be really useful within the existing > code base. > > Paddy > > Get Outlook for iOS > ________________________________ > From: Chao Sun <sunc...@apache.org> > Sent: Wednesday, November 21, 2018 2:42 PM > To: dev@arrow.apache.org > Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev > Subject: Re: [DISCUSS] Rust add adapter for parquet > > > A bigger issue is windows support for parquet-rs, Chao – I don’t believe > that parquet-rs supports windows, right? When I test it myself I get > errors regarding clang and libclang which parquet-rs must use. > > I think there are some issue regarding clang working with zstd on 32-bit > platform. However, it was able to compile with target: x86_64-pc-windows-msvc, > which seems to be the only one that arrow is using for windows CI. So I > think we can move forward and address the 32-bit platform issue later. > > On Wed, Nov 21, 2018 at 10:18 AM paddy horan <paddyho...@hotmail.com> wrote: > > > I think using nightly is fine for the reasons mentioned already. We > > should switch our CI to still run CI against stable (non fatal) so we know > > when we can make the move back to stable. > > > > > > > > A bigger issue is windows support for parquet-rs, Chao – I don’t believe > > that parquet-rs supports windows, right? When I test it myself I get > > errors regarding clang and libclang which parquet-rs must use. > > > > > > > > We have had windows support pretty much since the beginning for Rust. Is > > it possible to put parquet support behind a feature gate initially and only > > run CI for non-windows? I would be willing to help get windows support > > working after the fact, although I know very little about parquet right now. > > > > > > > > Are there other strategies for dealing with this? > > > > > > > > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for > > Windows 10 > > > > > > > > ________________________________ > > From: Chao Sun <sunc...@apache.org> > > Sent: Wednesday, November 21, 2018 12:52:32 PM > > To: Wes McKinney > > Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev > > Subject: Re: [DISCUSS] Rust add adapter for parquet > > > > > Can you remind us all why nightly is required? > > > > Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119> > > for all the unstable features parquet-rs uses. I'm personally inclined to > > use nightly since some new features such as specialization makes > > development much easier. Like Andy mentioned, as we are still developing > > arrow + parquet I think we can stay with nightly and transition back to > > stable once major features are implemented and more and more people start > > to use it (hopefully by that time the unstable features are stabilized). > > Moreover, I've seen quite a few popular projects rely on nightly such as > > rocket, tikv, etc., so seems it is not uncommon in the Rust world. > > > > > The steps from here are for you all to get the codebase into a state > > that is ready for donation, including ASF license headers, etc. A pull > > request into apache/arrow would be the best thing > > > > Sure. I'll prepare a pull request in the next few days, and then we can > > proceed to the voting, ICLA, etc. Thanks. > > > > Chao > > > > > > On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > The steps from here are for you all to get the codebase into a state > > > that is ready for donation, including ASF license headers, etc. A pull > > > request into apache/arrow would be the best thing > > > > > > Then we have to do the following > > > > > > * Vote on the Arrow mailing list > > > * Receive ICLAs from contributors > > > * Complete IP clearance > > > * Merge codebase > > > > > > Let me know when you are ready to move forward. From start to finish > > > that can get done in approximately 6 days if the code is ready > > > > > > Thanks > > > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <andygrov...@gmail.com> > > wrote: > > > > > > > > Renjie, > > > > > > > > Can you remind us all why nightly is required? > > > > > > > > My personal feeling is that stable is a nice-to-have, but Rust is still > > > moving fast and we are on the bleeding edge here so I'm OK with Arrow > > > relying on nightly for now. Maybe we can have a plan to transition back > > to > > > stable for a future release if we go with nightly now. > > > > > > > > Thanks, > > > > > > > > Andy. > > > > > > > > > > > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <liurenjie2...@gmail.com> > > > wrote: > > > >> > > > >> That sounds great. But parquet-rs currently relies on nightly rust, > > that > > > >> would be the first problem to resolve. > > > >> > > > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <andygrov...@gmail.com> > > > wrote: > > > >> > > > >> > This sounds like a great idea. > > > >> > > > > >> > With support for both CSV and Parquet in the Arrow crate, it would > > be > > > nice > > > >> > to design a standard interface for Arrow data sources. Maybe this is > > > as > > > >> > simple as implementing `Iterator<RecordBatch>`. > > > >> > > > > >> > Andy. > > > >> > > > > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <sunc...@apache.org> > > wrote: > > > >> > > > > >> > > Yes, we'd be interested to move forward. I'm inclined to merge > > this > > > into > > > >> > > Arrow because of the issues that you pointed out with parquet c++ > > > merge, > > > >> > > and I do see a tight relationship between the two projects, and > > > potential > > > >> > > sharing of common libraries. @Ivan Sadikov < > > ivan.sadi...@gmail.com> > > > what > > > >> > > do you think? > > > >> > > > > > >> > > Chao > > > >> > > > > > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney < > > wesmck...@gmail.com> > > > >> > wrote: > > > >> > > > > > >> > >> hi folks, > > > >> > >> > > > >> > >> Would you all be interested in moving forward the parquet-rs > > > project? > > > >> > >> I have a little more bandwidth to help with the code donation in > > > the > > > >> > >> next month or two. > > > >> > >> > > > >> > >> I know we voted on the Parquet mailing list about the donation > > > >> > >> already. One big question is whether you want to create an > > > >> > >> apache/parquet-rs repository or whether you want to co-develop > > > >> > >> parquet-rs together with Arrow in Rust, similar to what we are > > > doing > > > >> > >> with C++. It's possible you might run into the same kinds of > > issues > > > >> > >> that led us to consider the monorepo arrangement. > > > >> > >> > > > >> > >> Thanks > > > >> > >> Wes > > > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu < > > > liurenjie2...@gmail.com> > > > >> > >> wrote: > > > >> > >> > > > > >> > >> > Hi, Chao: > > > >> > >> > I've opened an jira issue for that and planning to work on > > that. > > > >> > >> > > > > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu < > > > liurenjie2...@gmail.com> > > > >> > >> wrote: > > > >> > >> > > > > >> > >> > > Yes, it's a mistake, sorry for that > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun < > > sunc...@apache.org> > > > >> > wrote: > > > >> > >> > > > > > >> > >> > >> (s/flink/arrow - it is a mistake?) > > > >> > >> > >> > > > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step > > in > > > >> > >> parquet-rs > > > >> > >> > >> is to integrate with Apache Arrow. Actually we just had a > > > >> > discussion > > > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about > > this > > > >> > >> recently. > > > >> > >> > >> Feel free to share your comments on the github. > > > >> > >> > >> > > > >> > >> > >> Best, > > > >> > >> > >> Chao > > > >> > >> > >> > > > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu < > > > >> > liurenjie2...@gmail.com > > > >> > >> > > > > >> > >> > >> wrote: > > > >> > >> > >> > > > >> > >> > >>> cc:Sunchao and Any > > > >> > >> > >>> > > > >> > >> > >>> > > > >> > >> > >>> ---------- Forwarded message --------- > > > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com> > > > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM > > > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet > > > >> > >> > >>> To: <dev@arrow.apache.org> > > > >> > >> > >>> > > > >> > >> > >>> > > > >> > >> > >>> Hello, > > > >> > >> > >>> > > > >> > >> > >>> you might also want to raise this with the > > > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap > > > >> > between > > > >> > >> the > > > >> > >> > >>> developers of this project and the Arrow Rust > > implementation > > > is > > > >> > >> quite large > > > >> > >> > >>> but still it may make sense to also start a discussion > > there. > > > >> > >> > >>> > > > >> > >> > >>> Uwe > > > >> > >> > >>> > > > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote: > > > >> > >> > >>> > Hi, all: > > > >> > >> > >>> > > > > >> > >> > >>> > Now the rust component is approaching a stable state and > > > rust > > > >> > >> reader > > > >> > >> > >>> for > > > >> > >> > >>> > parquet is ready. I think it maybe a good time to start > > an > > > >> > >> adapter for > > > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys > > > think > > > >> > >> about it? > > > >> > >> > >>> > -- > > > >> > >> > >>> > Liu, Renjie > > > >> > >> > >>> > Software Engineer, MVAD > > > >> > >> > >>> -- > > > >> > >> > >>> Liu, Renjie > > > >> > >> > >>> Software Engineer, MVAD > > > >> > >> > >>> > > > >> > >> > >> > > > >> > >> > >> -- > > > >> > >> > > Liu, Renjie > > > >> > >> > > Software Engineer, MVAD > > > >> > >> > > > > > >> > >> > -- > > > >> > >> > Liu, Renjie > > > >> > >> > Software Engineer, MVAD > > > >> > >> > > > >> > > > > > >> > > > > >> -- > > > >> Renjie Liu > > > >> Software Engineer, MVAD > > > > >
