Re: [VOTE][RUST] Release Apache Arrow Rust 17.0.0 RC1

+1 (binding) Verified on aarch64 macos On Sat, 25 Jun 2022 at 07:00, QP Hou wrote: > +1 (binding) > > On Fri, Jun 24, 2022 at 7:36 PM Remzi Yang <1371656737...@gmail.com> > wrote: > > > > +1 (non-binding). Verified on Mac M1. > > Thanks Andrew. > > > > Remzi > > > > On Sat, 25 Jun 2022 at 09:33

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 10.0.0 RC1

+1 (binding) There are 3 test failures because the hardcoded strings expect "ARROW_TEST_DATA" but we get "privateARROW_TEST_DATA" when verifying the RC. An example can be seen on this diff https://www.diffchecker.com/B8i8w4GC. My conclusion from looking at it is that it's likely a quirk that we

Re: [VOTE][RUST][DataFusion] Release Apache Arrow DataFusion 10.0.0 RC1

I've logged https://github.com/apache/arrow-datafusion/issues/2916 On Fri, 15 Jul 2022 at 08:34, Neville Dipale wrote: > +1 (binding) > > There are 3 test failures because the hardcoded strings expect > "ARROW_TEST_DATA" but we get "privateARROW_TEST_DATA&

Re: [VOTE][RUST] Release Apache Arrow Rust 25.0.0 RC1

+1 (binding) Verified on Ubuntu (WSL2) On Sun, 16 Oct 2022 at 13:50, Remzi Yang <1371656737...@gmail.com> wrote: > +1 (non-binding) Verified on M1 Mac. > > Thank you Andrew. > > On Sun, 16 Oct 2022 at 11:18, L. C. Hsieh wrote: > > > +1 (binding) > > > > Verified on Intel Mac. > > > > Thanks And

Re: [VOTE] Release Apache Arrow 10.0.0 - RC0

"Re: [VOTE] Release Apache Arrow 10.0.0 - RC0" on Sun, 23 Oct 2022 > 10:31:27 +0300, > Benson Muite wrote: > > > WIP but source verification fails for me on CentOS 7 due to unsigned > > key from Neville Dipale: > > > > TEST_DEFAULT=0 TEST_SOURCE=1 dev/rel

Re: [VOTE] Accept donation of Rust Ballista project

+1 (binding) On Sun, 21 Mar 2021 at 19:48, Neal Richardson wrote: > +1 (binding) > > Neal > > On Sun, Mar 21, 2021 at 10:38 AM Francis Du wrote: > > > I have been following Ballista for a long time. This is an exciting > thing. > > I am also very happy to contribute to this. > > > > So +1 > > >

Re: [VOTE] Move Rust components to new repos and process

Hi Arrow devs, +1 On Thu, 15 Apr 2021 at 05:28, Jorge Cardoso Leitão wrote: > +1 binding. > > On Thu, Apr 15, 2021 at 4:10 AM Micah Kornfield > wrote: > > > +1 binding. Does this also cover other changes (issue management) +1 to > > those as well. > > > > On Wednesday, April 14, 2021, Andy Gro

[DISCUSS] Moving the format directory to arrow-format repository

Hi Arrow devs, Andy noticed that we carry a copy of the format directory in arrow-rs, which is bound to get outdated in the future. We would like to propose creating an arrow-format repository, similar to parquet-format, so that arrow-rs and other future separate repositories could add this as a

Re: [DISCUSS] Moving the format directory to arrow-format repository

t files > > in > > > apache/arrow, even if there were a separate apache/arrow-format > > repository. > > > The format files are slow-moving enough that I don't think it's > > burdensome > > > to mirror these into satellite repositories like arro

Re: Delta Lake support for DataFusion

The correct approach might be to improve DataFusion support in delta-rs. TableProvider is already implemented here: https://github.com/delta-io/delta-rs/blob/main/rust/src/delta_datafusion.rs I've pinged QP to ask for their advice. Neville On Wed, 9 Jun 2021 at 19:58, Andrew Lamb wrote: > I th

Re: [ANNOUNCE] New Arrow PMC member: Neville Dipale

at 3:20 PM Wes McKinney > > > wrote: > > > > > > > > > > The Project Management Committee (PMC) for Apache Arrow has invited > > > > > Neville Dipale to become a PMC member and we are pleased to > announce > > > > > that Neville has accepted. > > > > > > > > > > Congratulations and welcome! > > > > > > > > >

Re: [VOTE][RUST] Release Apache Arrow Rust 5.1.0 RC1

+1 Ran ./dev/release/verify-release-candidate.sh 5.1.0 1 on aarch64-apple-darwin Got + TEST_SUCCESS=yes + echo 'Release candidate looks good!' Release candidate looks good! + exit 0 + cleanup On Fri, 30 Jul 2021 at 15:53, Wayne Xia wrote: > +1 > > I ran this on Intel macOS Catalina: > ./dev/r

Re: [VOTE][RUST] Release Apache Arrow Rust 5.2.0 RC1

+1 (binding) I ran the verification script on aarch64 macos On Fri, 13 Aug 2021 at 07:45, QP Hou wrote: > +1 (non-binding) > > ran the verification script on Linux 5.4.0 x86_64 > > On Thu, Aug 12, 2021 at 12:44 PM Andrew Lamb wrote: > > > > Hi, > > > > I would like to propose a release of Apac

Re: [VOTE][RUST][Datafusion] Release Apache Arrow Datafusion 5.0.0 RC3

+1 (bniding) i verified the RC on aarch64-macos On Fri, 13 Aug 2021 at 23:29, Jorge Cardoso Leitão wrote: > +1 > > Great work everyone! > > > On Fri, Aug 13, 2021, 22:19 Daniël Heres wrote: > > > +1 (non binding). Looking good. > > > > > > On Fri, Aug 13, 2021, 07:49 QP Hou wrote: > > > > > G

Re: VOTE][RUST] Release Apache Arrow Rust 8.0.0 RC1

Good day +1 (binding), verified on macos-aarch64 On Tue, 25 Jan 2022 at 05:08, Wang Xudong wrote: > +1(non-binding) > Verified on macos > "Release candidate looks good!" > > -- > xudong963 > > Andy Grove 于2022年1月25日周二 09:07写道： > > > +1 (binding). Verified on Ubuntu 20.04.3 LTS. > > > > On Fri,

Re: [VOTE][RUST] Release Apache Arrow Rust 9.0.3 RC3

[X] +0 binding There's a test failure on aarch64, I've opened https://github.com/apache/arrow-rs/issues/1294 thread 'util::bit_chunk_iterator::tests::test_unaligned_bit_chunk_iterator' panicked at 'assertion failed: ALIGNMENT > 64', arrow/src/util/bit_chunk_iterator.rs:470:9 On Wed, 9 Feb 2022 a

Re: [VOTE][RUST] Release Apache Arrow Rust 9.0.3 RC3

Sorry, that should be a -1 because of the bug On Thu, 10 Feb 2022 at 09:42, Neville Dipale wrote: > [X] +0 binding > > There's a test failure on aarch64, I've opened > https://github.com/apache/arrow-rs/issues/1294 > > thread > 'util::bit_chunk_iterator::test

Re: [VOTE][RUST] Release Apache Arrow Rust 9.0.3 RC3

tails can be found here [1] > > Andrew > > [1] https://github.com/apache/arrow-rs/pull/1297#issuecomment-1034888108 > > On Thu, Feb 10, 2022 at 2:43 AM Neville Dipale > wrote: > > > Sorry, that should be a -1 because of the bug > > > > On Thu, 10 Feb 2022 at

Re: [VOTE][RUST] Release Apache Arrow Rust 9.0.3 RC3

> More details can be found here [1] > >> > > >> > Andrew > >> > > >> > [1] > >> https://github.com/apache/arrow-rs/pull/1297#issuecomment-1034888108 > >> > > >> > On Thu, Feb 10, 2022 at 2:43 AM Neville Dipale > >

Re: [VOTE][RUST][Datafusion] Release Apache Arrow Datafusion 7.0.0 RC2

+1 binding, verified the RC on macos aarch64 On Tue, 15 Feb 2022 at 09:18, Yijie Shen wrote: > Thanks, Andrew! > > +1 non-binding > > > On Tue, Feb 15, 2022 at 11:51 AM QP Hou wrote: > > > +1 non-binding, went release script on Linux arm64. > > > > It failed at the end when executing the `cargo

Re: [ANNOUNCE] New Arrow PMC member: QP Hou

Congratulations QP! 🥳 On Fri, 18 Feb 2022 at 09:42, Daniël Heres wrote: > Congratulations QP! > > On Fri, Feb 18, 2022, 06:55 Benson Muite > wrote: > > > Congratulations QP! > > On 2/18/22 8:35 AM, Jiayu Liu wrote: > > > Congratulations QP! > > > > > > On Fri, Feb 18, 2022 at 1:32 PM Micah Korn

[VOTE][RUST] Release Apache Arrow Rust 9.1.0 RC1

Hi, I would like to propose a release of Apache Arrow Rust Implementation, version 9.1.0. This release candidate is based on commit: ecba7dc0830dbde6aa6dd9432519b776e40c1e85 [1] The proposed release tarball and signatures are hosted at [2]. The changelog is located at [3]. Please download, ver

Re: [VOTE][RUST] Release Apache Arrow Rust 9.1.0 RC1

x27; no = yes ']' > + echo 'Failed to verify release candidate. See /tmp/arrow-9.1.0.62KW4 for > details.' > > > On Sat, Feb 19, 2022 at 10:04 AM Neville Dipale > wrote: > > > Hi, > > > > I would like to propose a re

Re: [VOTE][RUST] Release Apache Arrow Rust 9.1.0 RC1

g: Signature made Sat 19 Feb 2022 10:01:41 AM MST > > gpg:using EDDSA key > 3905F254F9E504B40FFF6CF6000488D7717D3FB2 > > gpg: Can't check signature: No public key > > + cleanup > > + '[' no = yes ']' > > + echo 'Fail

Re: [VOTE][RUST] Release Apache Arrow Rust 9.1.0 RC1

+1 binding Verified on macos aarch64 + TEST_SUCCESS=yes + echo 'Release candidate looks good!' Release candidate looks good! On Sun, 20 Feb 2022 at 10:43, Neville Dipale wrote: > Hi Kou, > > Thanks, I've just managed to add my key through some fiddling before I

Re: [VOTE][RUST] Release Apache Arrow Rust 9.1.0 RC1

://dist.apache.org/repos/dist/release/arrow/KEYS > > > > ? > > > > > > > > The release/arrow/KEYS content is distributed as > > > > https://downloads.apache.org/arrow/KEYS . > > > > > > > > > > > > Thanks, > > > > -- &g

Re: [VOTE][RUST] Release Apache Arrow Rust 9.1.0 RC1

Thanks Andy, I didn't get invites for parquet, parquet-derive and arrow-flight On Tue, 22 Feb 2022 at 19:58, Andy Grove wrote: > Hi Neville, > > I have invited you to become an owner on the Arrow/DataFusion crates. > > Thanks, > > Andy. > > > > On Tue, F

Re: [VOTE][RUST] Release Apache Arrow Rust 9.1.0 RC1

; > On Tue, Feb 22, 2022 at 11:13 AM Neville Dipale > wrote: > > > Thanks Andy, I didn't get invites for parquet, parquet-derive and > > arrow-flight > > > > On Tue, 22 Feb 2022 at 19:58, Andy Grove wrote: > > > > > Hi Neville, > > >

Re: [VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release

+1 (non-binding) On Tue, 30 Jun 2020 at 06:29, Ben Kietzman wrote: > +1 (non binding) > > On Tue, Jun 30, 2020, 00:25 Wes McKinney wrote: > > > +1 (binding) > > > > On Mon, Jun 29, 2020 at 10:49 PM Micah Kornfield > > wrote: > > > > > > +1 (binding) > > > > > > On Mon, Jun 29, 2020 at 2:43 PM

Re: Bot to set "In Progress" status in JIRA

Thanks Wes, I noticed this today, but was a bit confused as to why you reassigned a JIRA to yourself, then back to me. This clarifies what happened :) Neville On Tue, 30 Jun 2020 at 15:39, Wes McKinney wrote: > hi, > > Yesterday I set up a bot to set issues to In Progress if they have an > ass

[Integration] Errors running archery integration on Windows

Hi Arrow devs, I'm trying to run archery integration tests in Windows 10 (Python 3.7.7; conda 4.8.3), but I'm getting an error *ValueError: low is out of bounds for int32* (https://gist.github.com/nevi-me/4946eabb2dc111e10b98c074b45b73b1 ). Has someone else encountered this problem before? Regar

Re: [Integration] Errors running archery integration on Windows

hevc a écrit : > > Numpy on windows has different default bitwidth than on linux. Perhaps > this > > is causing the issue? (see: > > > https://stackoverflow.com/questions/36278590/numpy-array-dtype-is-coming-as-int32-by-default-in-a-windows-10-64-bit-machine > > ) >

Status of Rust Integration Testing

Good day Arrow devs, I've spent a few evenings looking into the issues that we're experiencing with Rust integration testing. In summary, none of our tests pass (zero batch doesn't count :) ). This is mainly because of changes from the legacy padding in the 0.15.0 release, which we never made in R

Re: Status of Rust Integration Testing

pdate. One question, we have "gold" files for 0.14.0 > checked into the test-data repo and run integration tests on those to > ensure we can read them in a few implementations. Does Rust at least read > those correctly? > > Thanks, > Micah > > On Fri, Jul 10, 2020 at 1:

[Rust] Creating a separate branch for Arrow Partuet writer until next release

Good day, This relates to https://github.com/apache/arrow/pull/7319 (ARROW-8289) In the past few months we haven't had enough review bandwidth on Rust's Parquet implementation (mostly relying on Chao for non-trivial reviews), and given the amount of work needed for an Arrow writer + the interest

Re: Rust conversion

Hi Roland, For primitive types, there are value_slice methods which would allow you to get the array's contents as a vector, but you have to handle the null values as the vectors wouldn't return an Option. We haven't seen demand yet in converting arrays to JSON, but the integration crate ( https:/

[Rust] Arrow SQL Adapters/Connectors

Hi Arrow developers I would like to gauge the appetite for an Arrow SQL connector that: * Reads and writes Arrow data to and from SQL databases * Reads tables and queries into record batches, and writes batches to tables (either append or overwrite) * Leverages binary SQL formats where available

Re: [Rust] Arrow SQL Adapters/Connectors

that yields > > RecordBatches. The implementation of this trait by different providers > > would give support to be used in Arrow and DataFusion. > > > > Besides postgres, one idea is to pick the top from this list > > <https://db-engines.com/en/ranking>: > &

Re: 2.0.0 release timeline: October 9

Hi Neal, I've pruned the Rust backlog a bit, but only changed PRs that I've either opened, or those that I presume nobody is currently working on. There's 2 major features I've been working on: 1. Writing Arrow data to Parquet (separate branch) 2. Integration testing I'll prioritise 2 as that's

Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

+1 (non-binding) Rust support is behind, but we'll catch up at some point. On Wed, 30 Sep 2020 at 03:10, Holden Karau wrote: > +1 (non-binding) > > On Tue, Sep 29, 2020 at 6:08 PM Sutou Kouhei wrote: > > > +1 > > > > In > > "Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow > > sp

Re: [VOTE] Accept donation of Julia implementation for Apache Arrow

Good day 😊 +1 On Mon, 12 Oct 2020, 22:35 Neal Richardson, wrote: > Hi all, > Last month [1] Jacob Quinn proposed donating Arrow.jl, a Julia > implementation of Arrow, to the Apache Arrow project. The community has had > an opportunity to discuss this and there do not seem to be objections. > Th

[Rust] Merging the Parquet Arrow Branch

Hi Arrow devs, We've been working on a Parquet writer on a separate branch, mainly to expedite merging PRs in case there weren't enough reviewers.I wasn't comfortable merging the work by 2.0, so I opted not to get it merged for the release. There are people who have expressed an interest in using

Re: [ANNOUNCE] New Arrow committer: Andrew Lamb

Congrats Andrew! Look forward to working more with you On Tue, 10 Nov 2020, 18:07 Jorge Cardoso Leitão, wrote: > Congrats, Andrew! > > Andrew has been doing an amazing job, both on the implementation but also > at reviewing and helping others. He taught me a lot, I am having a great > time worki

Re: Arrow 3.0 release

Good day, I was hoping to complete the parquet list writer PR in time, but even though I've been burning the midnight oil addressing the remaining issues in the PR, I won't make it in time :( I've removed the Rust blocker from the milestone, so there should be nothing on my side blocking us. Tha

Re: Release 3.0 timeline?

Hi Arrow devs, There's some bugs in the Parquet implementation which affect reading of data: - https://issues.apache.org/jira/browse/ARROW-11269, which was opened today, and I just saw now. - an issue with list schema nulls from the parquet-format's logical types. In this case, we misinterpret th

Re: [VOTE] Release Apache Arrow 3.0.0 - RC2

This is my first time verifying, do I also need to set the env vars below? ARROW_GANDIVA=0 ARROW_PLASMA=0 TEST_DEFAULT=0 TEST_SOURCE=1 TEST_CPP=1 TEST_PYTHON=1 TEST_JAVA=1 TEST_INTEGRATION_CPP=1 TEST_INTEGRATION_JAVA=1 Otherwise, I'm currently running: ./dev/release/verify-release-candidate.bat 3

Re: [VOTE] Release Apache Arrow 3.0.0 - RC2

e errors from 2 runs. Neville On Fri, 22 Jan 2021 at 14:22, Neville Dipale wrote: > This is my first time verifying, do I also need to set the env vars below? > > ARROW_GANDIVA=0 ARROW_PLASMA=0 TEST_DEFAULT=0 > TEST_SOURCE=1 TEST_CPP=1 TEST_PYTHON=1 TEST_JAVA=1 > TEST_INTEGRATION_CPP=1 TE

Re: [VOTE] Release Apache Arrow 3.0.0 - RC2

ow C:\Users\nevi\Work\oss\arrow>EXIT /B 1 On Fri, 22 Jan 2021 at 15:14, Krisztián Szűcs wrote: > Thanks Neville for testing it! > > There should be more context about the failures above the summary. > Could you please post the errors? > > > On Fri, Jan 22, 2021 at 2:05 PM

Re: [RUST] Fields and schema metadata

We had to use the BTreeMap because a HashMap doesn't implement Hash, so can't be used in the Field. The easiest way to see this, is to replace it with a HashMap, and try compile the arrow crate. Neville On Sat, 06 Feb 2021, 13:50 Fernando Herrera, wrote: > Hi all, Is there a reason why the Fie

Re: [DISCUSS] [Rust] Adding support for Flight protocol

Good evening With support for testing against integration files now done, I've resumed work on the IPC reader. If I don't encounter trouble reading the existing files, I expect to be done with this work by the end of the weekend. I had taken the approach of one large PR to include all Rust-support

Re: [DISCUSS] [Rust] Adding support for Flight protocol

Tonic, and I believe > they might be interested in helping here too. > > Let me know how else I can help out with this effort. > > Andy. > > > > On Thu, Oct 17, 2019 at 2:42 PM Neville Dipale > wrote: > > > Good evening > > > > With support f

[Help Needed] Arrow IPC Reader in Rust

Hi Arrow developers, I'm "done" with the Arrow IPC Reader in Rust (for supported data types), but am having issues with reading some of the test data. Specifically, I've noticed that when reading the integration test data (primitve_generated), where I expect an array with 17 values, the arrow arra

Re: [DISCUSS] [Rust] Adding support for Flight protocol

ing > > out. Since, I guess most of my experience up to now is with Tonic > itself, I > > wanted to make myself available to you all for help or special > > implementations coming from the Tonic side. I am also willing to help > with > > the flight rust implement

Re: [Help Needed] Arrow IPC Reader in Rust

__ > From: paddy horan > Sent: Saturday, November 16, 2019 1:03 PM > To: dev@arrow.apache.org > Subject: Re: [Help Needed] Arrow IPC Reader in Rust > > Hey Neville, > > I'll take a look if no-one beats me to it (I might not have time today or >

Re: [Discuss][Rust] Policy regarding "unsafe"

Hi Paddy, On Fri, 10 Jan 2020 at 04:13, paddy horan wrote: > Hi All, > > This time last year there was a brief discussion on the usage of unsafe in > Rust (a user on github raised the issue and I created the JIRA). [1] > > So far we mostly avoid unsafe in the public API's. The thinking here is

Re: [Discuss][Rust] Policy regarding "unsafe"

Hi Paddy, Arrow Developers, I've given this some thought, and I preliminarily think that perhaps we can audit our use of unsafe and evaluate where we can remove it, propagate it upwards (and provide safe alternatives) or provide some safety to callers. Looking at the 3 options that Paul Kernfeld

Re: [Format] Make fields required?

I also like the idea of marking some fields as required, also erring on the side of conservatism. I can test on the Rust implementation (although IPC support is still incomplete) if we go ahead with this. On Fri, 17 Jan 2020 at 08:29, Micah Kornfield wrote: > I too, couldn't find anything that

Re: [Rust] Possible blocking issue for 0.16 release

Hi Andy, I think `cargo update` is the correct approach to resolve this issue. Yes, we've got quite a bit of backlog in Rust, but we should initiate the process to adopt parquet-format with Chao and Ivan's approval post 0.16. In the interim we could fix the thrift dependency issue upstream (as th

Re: [DISCUSS] Adding "trivial" buffer compression option to IPC protocol (ARROW-300)

I also support compression at the buffer level, and making it an extra message. Talking about compression and flight, has anyone tested using grpc's compression to compress at the transport level (if that's a correct way to describe it)? I believe only gzip and brotli are currently supported, so t

Re: [Gandiva] Representing logical query plans in protobuf

Hi Andy & Wes, Apologies if I go off-topic a bit, I hope my thoughts are related though. I'm a new contributor to Arrow, but I've been using and following it since the feather days. I'm interested in contributing to Rust, as that aligns more with my day job(s). I think we (rather, the Rust contr

Re: Arrow Rust roadmapping [was Re: [Gandiva] Representing logical query plans in protobuf]

inney wrote: > hi Neville, > > On Sat, Jan 5, 2019 at 2:37 PM Neville Dipale > wrote: > > > > Hi Andy & Wes, > > > > Apologies if I go off-topic a bit, I hope my thoughts are related though. > > > > I'm a new contributor to Arrow, but I&#x

[Rust] Creating Native Types from Rust Types (e.g. i32 -> ArrowNativeType)

I'm trying to find/create a way of converting primitive rust values in Rust to their ArrowNativeType counterparts. I'm trying to implement some array ops which need constant values, so I can be able to multiply say a Int32Type by 3 to get a value. Is there currently a way of doing this that I don'

Re: [Rust] Creating Native Types from Rust Types (e.g. i32 -> ArrowNativeType)

ooking for. On Mon, 07 Jan 2019, 17:09 Andy Grove It is possible to create an array of Arrow types from a vec of primitives > e.g. Int32Array::from(vec![1, 2, 3]) IIRC. > > Andy. > > On Mon, Jan 7, 2019 at 5:09 AM Neville Dipale > wrote: > > > I'm trying to find

Re: [Rust] Creating Native Types from Rust Types (e.g. i32 -> ArrowNativeType)

on arrays. On Mon, 7 Jan 2019 at 18:42, Neville Dipale wrote: > Thanks Andy, > > I'm aware of that, and have tried it with no success. What I am looking > for might be similar to ScalarValue in DataFusion. > > If I create an array like Int32Array::from(vec![1]), and try to

Re: [Rust] [DISCUSS] Donate DataFusion to Arrow project

Hi Andy, I can't comment on the voting process, but regarding the addition of DataFusion: I support the idea to donate the code, mainly as I think that will help us accelerate some work on Rust. Out of curiousity, I've been prototying a 'Rust dataframe' abstraction which (can/will) have various s

Re: [VOTE] Accept donation of Rust DataFusion library for Apache Arrow

Hi Andy, +1 : Accept contribution of DataFusion Rust library Thanks On Wed, 23 Jan 2019 at 03:05, Wes McKinney wrote: > Dear all, > > The developers of DataFusion, an analytical query engine written > in Rust, based on the Arrow columnar memory format, are proposing > to donate the code to Apa

Re: [Rust] code style: restrict line width to 90 characters?

Hi Chao, What's the current limit? I just ran rustfmt, and seems like it's not reformatting at 100 characters. I support changing whatever the current width is to 90 characters. Regards Neville On Fri, 25 Jan 2019 at 19:49, Chao Sun wrote: > Hi Rust developers, > > Just want to know if anyone

[Format] [Rust] ChunkedArray, Column and Table

Hi Arrow developers, I've been looking at the various language impls, and although a Table isn't currently part of the spec, it seems to be implemented in CPP, Python, Go, JS (and perhaps other languages). Are there plans of standardising these and adding them to the spec? I'm asking because I'm

Re: [Format] [Rust] ChunkedArray, Column and Table

t; (reading that spec though, it's not obvious to me why the Record Batch > definition doesn't reference a Schema) > > Regards > > Antoine. > > > Le 25/01/2019 à 19:48, Neville Dipale a écrit : > > Hi Arrow developers, > > > > I've been looking at t

Re: [Rust] Rust 0.13.0 release

Thanks for bringing this up Andy. I'm unemployed/on recovery leave, so I've had some surplus time to work on Rust. There's a lot of features that I've wanted to work on, some which I've spent some time attempting, but struggled with. A few block additional work that I could contribute. In 0.13.0

Re: [ANNOUNCE] New Arrow committer: Chao Sun

Congratulations Chao and Paddy! I'm loving the increase in velocity on the Rust side On Thu, 28 Feb 2019, 17:17 Wes McKinney, wrote: > thank you Chao, and welcome! > > On Thu, Feb 28, 2019 at 6:18 AM paddy horan > wrote: > > > > Congrats Chao! > > > > Get Outlook for iOS

Re: [Rust] [DataFusion] Preferences on futures / threading crates?

I'm a fan of using Rayon. Perhaps if it's not too much work, we could compare the two On Mon, 4 Mar 2019 at 15:04, Krisztián Szűcs wrote: > On Mon, Mar 4, 2019 at 5:55 AM Andy Grove wrote: > > > I have been working on a PoC of parallel query execution and it is > working > > well, and I am now

[Discuss] Array Cast Kernels Support Matrix

Hi Arrow devs, I'm currently adding support for casting arrays in Rust, and I'm wondering what casting operations should be supported, and how. Most operations are simple, but I have a few questions below. * Struct to Struct: I am not supporting in Rust as it might not make sense/be easy to suppo

Re: Timeline for 0.13 Arrow release

Hi Wes, In Rust, we have 2 bugs (https://issues.apache.org/jira/browse/ARROW-4914, https://issues.apache.org/jira/browse/ARROW-4886) both related to array slicing. In summary: * ARROW-4914, the bitmask of the original array is used to determine the validity of the sliced array, but offsets aren'

Re: Timeline for 0.13 Arrow release

etails in the JIRAs (e.g., test case to reproduce). Array currently > doesn't expose a bitmask API, and I don't think we need specialized > implementations for struct & list. > > Chao > > On Mon, Mar 18, 2019 at 8:24 PM Neville Dipale > wrote: > > > Hi Wes

Re: Timeline for 0.13 Arrow release

When is the cut-off for PRs? We have a public holiday on Thursday, and I want to use that to finish off my work on array casting. If that'll be too late I can defer to the next release. On Tue, 19 Mar 2019, 12:34 Antoine Pitrou, wrote: > > The only potential blocker from my POV is > https://iss

Re: Proper way to retrigger Travis CI builds

To add here, sometimes builds for unrelated changes are caused by your branch being behind master. I've noticed that whenever I rebase my changes to latest master, I reliably only trigger the Rust jobs to run. Maybe that could also help non-Arrow commiters :) On Thu, 25 Apr 2019 at 14:11, Wes McKi

[Rust] [Format] Should Null Bitmaps be Padded to 8 or 64 Bits?

Hi Arrow developers, I'm currently working on IPC in Rust, specifically reading Arrow files. I've noticed that null buffers/bitmaps are always padded to 64 bits (from pyarrow, not sure about others), while in Rust we pad to 8 bits. 1. Is this fine re. Rust per the spec? I'm having issues with re

Re: [ANNOUNCE] New Arrow committer: Neville Dipale

turday, May 11, 2019, paddy horan > > > > > wrote: > > > > > > > > > > > > > > > Congrats Neville! Thank you for your contributions! > > > > > > > > > > > > > > > > Get Outlook for iOS<https://aka.ms/

[Rust] [Discuss] Generalising Eq, Neq Kernel Functions Beyond Numeric Types

Hi Arrow[Rust] developers, I came across an instance where I wanted to compare 2 arrays that aren't numeric (bool, string, list?), and couldn't conveniently leverage the comparison array_ops for this. This is due to the trait bounds that require that PrimitiveArray satisfy T: ArrowNumericType. Us

[jira] [Created] (ARROW-8881) [Rust] Add large list and binary support

Neville Dipale created ARROW-8881: - Summary: [Rust] Add large list and binary support Key: ARROW-8881 URL: https://issues.apache.org/jira/browse/ARROW-8881 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-8883) [Rust] [Integration Testing] Disable unsupported tests

Neville Dipale created ARROW-8883: - Summary: [Rust] [Integration Testing] Disable unsupported tests Key: ARROW-8883 URL: https://issues.apache.org/jira/browse/ARROW-8883 Project: Apache Arrow

[jira] [Created] (ARROW-9007) [Rust] Support appending arrays by merging array data

Neville Dipale created ARROW-9007: - Summary: [Rust] Support appending arrays by merging array data Key: ARROW-9007 URL: https://issues.apache.org/jira/browse/ARROW-9007 Project: Apache Arrow

[jira] [Created] (ARROW-9053) [Rust] Add sort for lists and structs

Neville Dipale created ARROW-9053: - Summary: [Rust] Add sort for lists and structs Key: ARROW-9053 URL: https://issues.apache.org/jira/browse/ARROW-9053 Project: Apache Arrow Issue Type: Sub

[jira] [Created] (ARROW-9095) [Rust] Fix NullArray to comply with spec

Neville Dipale created ARROW-9095: - Summary: [Rust] Fix NullArray to comply with spec Key: ARROW-9095 URL: https://issues.apache.org/jira/browse/ARROW-9095 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-6650) [Rust] [Integration] Add method to generate JSON from RecordBatch

Neville Dipale created ARROW-6650: - Summary: [Rust] [Integration] Add method to generate JSON from RecordBatch Key: ARROW-6650 URL: https://issues.apache.org/jira/browse/ARROW-6650 Project: Apache

[jira] [Created] (ARROW-6928) [Rust] Add FixedSizeList type

Neville Dipale created ARROW-6928: - Summary: [Rust] Add FixedSizeList type Key: ARROW-6928 URL: https://issues.apache.org/jira/browse/ARROW-6928 Project: Apache Arrow Issue Type: Sub-task

[jira] [Created] (ARROW-6944) [Rust] Add StringType

Neville Dipale created ARROW-6944: - Summary: [Rust] Add StringType Key: ARROW-6944 URL: https://issues.apache.org/jira/browse/ARROW-6944 Project: Apache Arrow Issue Type: Sub-task

[jira] [Created] (ARROW-7193) [Rust] Create Arrow stream reader

Neville Dipale created ARROW-7193: - Summary: [Rust] Create Arrow stream reader Key: ARROW-7193 URL: https://issues.apache.org/jira/browse/ARROW-7193 Project: Apache Arrow Issue Type: Sub

[jira] [Created] (ARROW-7194) [Rust] CSV Writer causing recursion errors

Neville Dipale created ARROW-7194: - Summary: [Rust] CSV Writer causing recursion errors Key: ARROW-7194 URL: https://issues.apache.org/jira/browse/ARROW-7194 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-7207) [Rust] Update Generated Flatbuffer Files

Neville Dipale created ARROW-7207: - Summary: [Rust] Update Generated Flatbuffer Files Key: ARROW-7207 URL: https://issues.apache.org/jira/browse/ARROW-7207 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-7324) [Rust] Add Timezone to Timestamp

Neville Dipale created ARROW-7324: - Summary: [Rust] Add Timezone to Timestamp Key: ARROW-7324 URL: https://issues.apache.org/jira/browse/ARROW-7324 Project: Apache Arrow Issue Type: Sub-task

[jira] [Created] (ARROW-7364) [Rust] Add cast options to cast kernel

Neville Dipale created ARROW-7364: - Summary: [Rust] Add cast options to cast kernel Key: ARROW-7364 URL: https://issues.apache.org/jira/browse/ARROW-7364 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-7460) [Rust] Improve arithmetic kernels with autovec

Neville Dipale created ARROW-7460: - Summary: [Rust] Improve arithmetic kernels with autovec Key: ARROW-7460 URL: https://issues.apache.org/jira/browse/ARROW-7460 Project: Apache Arrow Issue

[jira] [Created] (ARROW-7475) [Rust] Create Arrow Stream writer

Neville Dipale created ARROW-7475: - Summary: [Rust] Create Arrow Stream writer Key: ARROW-7475 URL: https://issues.apache.org/jira/browse/ARROW-7475 Project: Apache Arrow Issue Type: Sub

[jira] [Created] (ARROW-7521) [Rust] Remove tuple on FixedSizeList datatype

Neville Dipale created ARROW-7521: - Summary: [Rust] Remove tuple on FixedSizeList datatype Key: ARROW-7521 URL: https://issues.apache.org/jira/browse/ARROW-7521 Project: Apache Arrow Issue

[jira] [Created] (ARROW-7620) [Rust] Windows builds failing due to flatbuffer compile error

Neville Dipale created ARROW-7620: - Summary: [Rust] Windows builds failing due to flatbuffer compile error Key: ARROW-7620 URL: https://issues.apache.org/jira/browse/ARROW-7620 Project: Apache Arrow

[jira] [Created] (ARROW-7704) [Rust] Support sort

Neville Dipale created ARROW-7704: - Summary: [Rust] Support sort Key: ARROW-7704 URL: https://issues.apache.org/jira/browse/ARROW-7704 Project: Apache Arrow Issue Type: New Feature

[jira] [Created] (ARROW-7705) [Rust] Initial sort implementation

Neville Dipale created ARROW-7705: - Summary: [Rust] Initial sort implementation Key: ARROW-7705 URL: https://issues.apache.org/jira/browse/ARROW-7705 Project: Apache Arrow Issue Type: Sub

[jira] [Created] (ARROW-7924) [Rust] Add sort for float types

Neville Dipale created ARROW-7924: - Summary: [Rust] Add sort for float types Key: ARROW-7924 URL: https://issues.apache.org/jira/browse/ARROW-7924 Project: Apache Arrow Issue Type: Sub-task

  1   2   >