Re: Need 64-bit Integer length for Parquet ByteArray Type

2019-04-05 Thread Ryan Blue
th fields all over the > place. > > External file references to BLOBS is doable but not the elegant, > integrated solution I was hoping for. > > -Brian > > On Apr 5, 2019, at 1:53 PM, Ryan Blue wrote: > > *EXTERNAL* > Looks like we will need a new encoding for this:

Re: Need 64-bit Integer length for Parquet ByteArray Type

2019-04-05 Thread Ryan Blue
thrift.h > > inline void DeserializeThriftMsg(const uint8_t* buf, uint32_t* len, T* > deserialized_msg) { > inline int64_t SerializeThriftMsg(T* obj, uint32_t len, OutputStream* out) > > -Brian > > On 4/5/19, 1:32 PM, "Ryan Blue" wrote: > > EXTERNAL > >

Re: Need 64-bit Integer length for Parquet ByteArray Type

2019-04-05 Thread Ryan Blue
at > require file format versioning changes? > > I realize this a non-trivial ask. Thanks for considering it. > > -Brian > -- Ryan Blue Software Engineer Netflix

Re: Developing a "dataset" API / framework for Arrow C++ users

2019-02-27 Thread Ryan Blue
t; > > > > > This project will also allow for "user-defined" data sources, so that > > > other people in the Arrow ecosystem can contribute new data interfaces > > > to interact with different kinds of storage systems using a common > > > API, so if they want to "plug in" to any computation layers available > > > in Apache Arrow then there is a reasonably straightforward path to do > > > that. > > > > > > Your comments and ideas on this project would be appreciated. > > > > > > Thank you, > > > Wes > > > > > > -- Ryan Blue Software Engineer Netflix

Re: [DISCUSS] Solutions for improving the Arrow-Parquet C++ development morass

2018-08-07 Thread Ryan Blue
gt;>> codebase. > >> >>> >> That > >> >>> >> >>> >> gives me hope that the projects could be managed > separately > >&g