I was using x86_64-pc-windows-msvc but it was just a quick test, I’ll take a closer look when I get a chance. I agree that lack of support for 32 bit should not hold this up.
Is the change to the nightly compiler the kind of thing that the PMC should vote on? This could be done in advance of the code donantion. Specialization, in particular, would be really useful within the existing code base. Paddy Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Chao Sun <sunc...@apache.org> Sent: Wednesday, November 21, 2018 2:42 PM To: dev@arrow.apache.org Cc: Wes McKinney; Andy Grove; Ivan Sadikov; Parquet Dev Subject: Re: [DISCUSS] Rust add adapter for parquet > A bigger issue is windows support for parquet-rs, Chao – I don’t believe that parquet-rs supports windows, right? When I test it myself I get errors regarding clang and libclang which parquet-rs must use. I think there are some issue regarding clang working with zstd on 32-bit platform. However, it was able to compile with target: x86_64-pc-windows-msvc, which seems to be the only one that arrow is using for windows CI. So I think we can move forward and address the 32-bit platform issue later. On Wed, Nov 21, 2018 at 10:18 AM paddy horan <paddyho...@hotmail.com> wrote: > I think using nightly is fine for the reasons mentioned already. We > should switch our CI to still run CI against stable (non fatal) so we know > when we can make the move back to stable. > > > > A bigger issue is windows support for parquet-rs, Chao – I don’t believe > that parquet-rs supports windows, right? When I test it myself I get > errors regarding clang and libclang which parquet-rs must use. > > > > We have had windows support pretty much since the beginning for Rust. Is > it possible to put parquet support behind a feature gate initially and only > run CI for non-windows? I would be willing to help get windows support > working after the fact, although I know very little about parquet right now. > > > > Are there other strategies for dealing with this? > > > > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for > Windows 10 > > > > ________________________________ > From: Chao Sun <sunc...@apache.org> > Sent: Wednesday, November 21, 2018 12:52:32 PM > To: Wes McKinney > Cc: Andy Grove; dev@arrow.apache.org; Ivan Sadikov; Parquet Dev > Subject: Re: [DISCUSS] Rust add adapter for parquet > > > Can you remind us all why nightly is required? > > Here's a tracking issue <https://github.com/sunchao/parquet-rs/issues/119> > for all the unstable features parquet-rs uses. I'm personally inclined to > use nightly since some new features such as specialization makes > development much easier. Like Andy mentioned, as we are still developing > arrow + parquet I think we can stay with nightly and transition back to > stable once major features are implemented and more and more people start > to use it (hopefully by that time the unstable features are stabilized). > Moreover, I've seen quite a few popular projects rely on nightly such as > rocket, tikv, etc., so seems it is not uncommon in the Rust world. > > > The steps from here are for you all to get the codebase into a state > that is ready for donation, including ASF license headers, etc. A pull > request into apache/arrow would be the best thing > > Sure. I'll prepare a pull request in the next few days, and then we can > proceed to the voting, ICLA, etc. Thanks. > > Chao > > > On Wed, Nov 21, 2018 at 7:55 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > The steps from here are for you all to get the codebase into a state > > that is ready for donation, including ASF license headers, etc. A pull > > request into apache/arrow would be the best thing > > > > Then we have to do the following > > > > * Vote on the Arrow mailing list > > * Receive ICLAs from contributors > > * Complete IP clearance > > * Merge codebase > > > > Let me know when you are ready to move forward. From start to finish > > that can get done in approximately 6 days if the code is ready > > > > Thanks > > On Wed, Nov 21, 2018 at 9:45 AM Andy Grove <andygrov...@gmail.com> > wrote: > > > > > > Renjie, > > > > > > Can you remind us all why nightly is required? > > > > > > My personal feeling is that stable is a nice-to-have, but Rust is still > > moving fast and we are on the bleeding edge here so I'm OK with Arrow > > relying on nightly for now. Maybe we can have a plan to transition back > to > > stable for a future release if we go with nightly now. > > > > > > Thanks, > > > > > > Andy. > > > > > > > > > On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu <liurenjie2...@gmail.com> > > wrote: > > >> > > >> That sounds great. But parquet-rs currently relies on nightly rust, > that > > >> would be the first problem to resolve. > > >> > > >> On Wed, Nov 21, 2018 at 4:49 AM Andy Grove <andygrov...@gmail.com> > > wrote: > > >> > > >> > This sounds like a great idea. > > >> > > > >> > With support for both CSV and Parquet in the Arrow crate, it would > be > > nice > > >> > to design a standard interface for Arrow data sources. Maybe this is > > as > > >> > simple as implementing `Iterator<RecordBatch>`. > > >> > > > >> > Andy. > > >> > > > >> > On Tue, Nov 20, 2018 at 11:46 AM Chao Sun <sunc...@apache.org> > wrote: > > >> > > > >> > > Yes, we'd be interested to move forward. I'm inclined to merge > this > > into > > >> > > Arrow because of the issues that you pointed out with parquet c++ > > merge, > > >> > > and I do see a tight relationship between the two projects, and > > potential > > >> > > sharing of common libraries. @Ivan Sadikov < > ivan.sadi...@gmail.com> > > what > > >> > > do you think? > > >> > > > > >> > > Chao > > >> > > > > >> > > On Tue, Nov 20, 2018 at 10:23 AM Wes McKinney < > wesmck...@gmail.com> > > >> > wrote: > > >> > > > > >> > >> hi folks, > > >> > >> > > >> > >> Would you all be interested in moving forward the parquet-rs > > project? > > >> > >> I have a little more bandwidth to help with the code donation in > > the > > >> > >> next month or two. > > >> > >> > > >> > >> I know we voted on the Parquet mailing list about the donation > > >> > >> already. One big question is whether you want to create an > > >> > >> apache/parquet-rs repository or whether you want to co-develop > > >> > >> parquet-rs together with Arrow in Rust, similar to what we are > > doing > > >> > >> with C++. It's possible you might run into the same kinds of > issues > > >> > >> that led us to consider the monorepo arrangement. > > >> > >> > > >> > >> Thanks > > >> > >> Wes > > >> > >> On Sun, Aug 19, 2018 at 11:11 PM Renjie Liu < > > liurenjie2...@gmail.com> > > >> > >> wrote: > > >> > >> > > > >> > >> > Hi, Chao: > > >> > >> > I've opened an jira issue for that and planning to work on > that. > > >> > >> > > > >> > >> > On Mon, Aug 20, 2018 at 11:03 AM Renjie Liu < > > liurenjie2...@gmail.com> > > >> > >> wrote: > > >> > >> > > > >> > >> > > Yes, it's a mistake, sorry for that > > >> > >> > > > > >> > >> > > > > >> > >> > > On Mon, Aug 20, 2018 at 10:57 AM Chao Sun < > sunc...@apache.org> > > >> > wrote: > > >> > >> > > > > >> > >> > >> (s/flink/arrow - it is a mistake?) > > >> > >> > >> > > >> > >> > >> Thanks Renjie for your interest. Yes, one of the next step > in > > >> > >> parquet-rs > > >> > >> > >> is to integrate with Apache Arrow. Actually we just had a > > >> > discussion > > >> > >> > >> <https://github.com/sunchao/parquet-rs/issues/140> about > this > > >> > >> recently. > > >> > >> > >> Feel free to share your comments on the github. > > >> > >> > >> > > >> > >> > >> Best, > > >> > >> > >> Chao > > >> > >> > >> > > >> > >> > >> On Sun, Aug 19, 2018 at 7:39 PM, Renjie Liu < > > >> > liurenjie2...@gmail.com > > >> > >> > > > >> > >> > >> wrote: > > >> > >> > >> > > >> > >> > >>> cc:Sunchao and Any > > >> > >> > >>> > > >> > >> > >>> > > >> > >> > >>> ---------- Forwarded message --------- > > >> > >> > >>> From: Uwe L. Korn <uw...@xhochy.com> > > >> > >> > >>> Date: Sun, Aug 19, 2018 at 5:08 PM > > >> > >> > >>> Subject: Re: [DISCUSS] Rust add adapter for parquet > > >> > >> > >>> To: <dev@arrow.apache.org> > > >> > >> > >>> > > >> > >> > >>> > > >> > >> > >>> Hello, > > >> > >> > >>> > > >> > >> > >>> you might also want to raise this with the > > >> > >> > >>> https://github.com/sunchao/parquet-rs project. The overlap > > >> > between > > >> > >> the > > >> > >> > >>> developers of this project and the Arrow Rust > implementation > > is > > >> > >> quite large > > >> > >> > >>> but still it may make sense to also start a discussion > there. > > >> > >> > >>> > > >> > >> > >>> Uwe > > >> > >> > >>> > > >> > >> > >>> On Thu, Aug 16, 2018, at 9:14 AM, Renjie Liu wrote: > > >> > >> > >>> > Hi, all: > > >> > >> > >>> > > > >> > >> > >>> > Now the rust component is approaching a stable state and > > rust > > >> > >> reader > > >> > >> > >>> for > > >> > >> > >>> > parquet is ready. I think it maybe a good time to start > an > > >> > >> adapter for > > >> > >> > >>> > parquet, just like adapter for orc in cpp. How you guys > > think > > >> > >> about it? > > >> > >> > >>> > -- > > >> > >> > >>> > Liu, Renjie > > >> > >> > >>> > Software Engineer, MVAD > > >> > >> > >>> -- > > >> > >> > >>> Liu, Renjie > > >> > >> > >>> Software Engineer, MVAD > > >> > >> > >>> > > >> > >> > >> > > >> > >> > >> -- > > >> > >> > > Liu, Renjie > > >> > >> > > Software Engineer, MVAD > > >> > >> > > > > >> > >> > -- > > >> > >> > Liu, Renjie > > >> > >> > Software Engineer, MVAD > > >> > >> > > >> > > > > >> > > > >> -- > > >> Renjie Liu > > >> Software Engineer, MVAD > > >
