Re: Wrap Unwarp Scalars in Cython API

2021-03-22 Thread Vibhatha Abeykoon
The use case is to pass a Scalar created in Python to a kernel written in C++ backend which supports arrow data types. To support this I need to unwrap the Pyarrow Scalar to a C++ arrow Scalar. With Regards, Vibhatha Abeykoon On Mon, Mar 22, 2021 at 11:15 PM Benjamin Kietzman wrote: > I'm not

Re: [C++] Obtain shared_ptr of an Array from a reference

2021-03-22 Thread Benjamin Kietzman
Would MakeArray(array.data()) work for you? On Mon, Mar 22, 2021, 23:00 Ying Zhou wrote: > Hi, > > I know this is a very silly question here but I still prefer to see it > resolved rather than working on it for a day: > > How shall I generate an std::shared_ptr from an Array&? Just taking > the

Re: Wrap Unwarp Scalars in Cython API

2021-03-22 Thread Benjamin Kietzman
I'm not sure what kind of unwrapping you are looking for, would pyarrow.scalar and Scalar.as_py address your use case? For example, pa.scalar(128) will wrap that integer into a Scalar On Mon, Mar 22, 2021, 11:15 Vibhatha Abeykoon wrote: > Hello, > > Is there a way to wrap and unwrap Scalars

[C++] Obtain shared_ptr of an Array from a reference

2021-03-22 Thread Ying Zhou
Hi, I know this is a very silly question here but I still prefer to see it resolved rather than working on it for a day: How shall I generate an std::shared_ptr from an Array&? Just taking the address and constructing a shared_ptr from the pointer doesn’t work. Ying

Re: [Java] Source control of generated flatbuffers code

2021-03-22 Thread Antoine Pitrou
Le 22/03/2021 à 20:17, bobtins a écrit : TL;DR: The Java implementation doesn't have generated flatbuffers code under source control, and the code generation depends on an unofficially-maintained Maven artifact. Other language implementations do check in the generated code; would it make sense

[Java] Source control of generated flatbuffers code

2021-03-22 Thread bobtins
TL;DR: The Java implementation doesn't have generated flatbuffers code under source control, and the code generation depends on an unofficially-maintained Maven artifact. Other language implementations do check in the generated code; would it make sense for this to be done for Java as well? I'm

Re: [DISCUSS] Revisiting LZ4 Compression for Arrow Buffers

2021-03-22 Thread Benjamin Wilhelm
> > Could you share the benchmark code/how the benchmark was run (does this > account for JIT warm-up time)? I just used the benchmark by the aircompressor project. They run the benchmark for a lot of algorithms on a lot of datasets so I commented out some to get faster results. You can find my

Re: [DISCUSS] Revisiting LZ4 Compression for Arrow Buffers

2021-03-22 Thread Micah Kornfield
> > I executed some of the benchmarks in the airlift/aircompressor project. I > found that aircompressior achieves on average only about 72% > throughput compared to the current version of the lz4-java JNI bindings > when compressing. When decompressing the gap is even bigger with around 56% >

Wrap Unwarp Scalars in Cython API

2021-03-22 Thread Vibhatha Abeykoon
Hello, Is there a way to wrap and unwrap Scalars using the Cython API? I am following the docs: https://arrow.apache.org/docs/python/extending.html But I couldn't find an option. Not sure if I am following the correct docs. With Regards, Vibhatha Abeykoon, PhD Candidate | Research Assistant,

Re: [DISCUSS] Revisiting LZ4 Compression for Arrow Buffers

2021-03-22 Thread Antoine Pitrou
Le 22/03/2021 à 15:29, Benjamin Wilhelm a écrit : Also, I would like to resume the discussion about the Frame format vs the Block format. There were 3 points for the Frame format by Antoine: - it allows streaming compression and decompression (meaning you can avoid loading a huge compressed

Re: [DISCUSS] Revisiting LZ4 Compression for Arrow Buffers

2021-03-22 Thread Benjamin Wilhelm
I executed some of the benchmarks in the airlift/aircompressor project. I found that aircompressior achieves on average only about 72% throughput compared to the current version of the lz4-java JNI bindings when compressing. When decompressing the gap is even bigger with around 56% throughout. See

Re: [VOTE] Accept donation of Rust Ballista project

2021-03-22 Thread Andrew Lamb
+1 On Sun, Mar 21, 2021 at 7:08 PM paddy horan wrote: > +1 (non-binding) > > > > From: Sutou Kouhei > Sent: Sunday, March 21, 2021 4:34:43 PM > To: dev@arrow.apache.org > Subject: Re: [VOTE] Accept donation of Rust Ballista project > > +1 (binding) > > In >

[NIGHTLY] Arrow Build Report for Job nightly-2021-03-22-0

2021-03-22 Thread Crossbow
Arrow Build Report for Job nightly-2021-03-22-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-22-0 Failed Tasks: - conda-linux-gcc-py36-aarch64: URL: